diff --git a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1a/result_protoemds_final_stack.md b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1a/result_protoemds_final_stack.md index 5b92fcb0..70ca47b3 100644 --- a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1a/result_protoemds_final_stack.md +++ b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1a/result_protoemds_final_stack.md @@ -17,21 +17,22 @@ This section identifies the technical context in which the protoEMDS Final Stack | Existing test ID | `2.2.3.1A` | | Level | UC + Technical | | ISO/IEC 25010 mapping | Functional suitability, Compatibility | -| Owner (tentative) | Alessio (Cefriel) / Wilhelm | +| Owner | Casper (imec, @vghelu49) | | Reviewer | Alessio | -| Deployment model assessed | TBD | -| Target environment | TBD | -| EDC version / release | TBD | -| Connector deployment reference | TBD | -| Assessment evidence | TBD | +| Deployment model assessed | CaaS / IONOS-managed deployment | +| Target environment | IONOS-managed CaaS deployment | +| EDC version / release | `emds-edc-connector` `9916cc5`, based on Eclipse EDC `0.10.0` | +| Connector deployment reference | `deployEMDS-k8s-deployment`, branch `prepare-prod`, commit `846e5f1` | +| Assessment evidence | Bruno CLI API execution and Playwright UI verification | The assessment should be completed using the deployment model for which evidence is realistically available. It is not mandatory to execute the same test in both CaaS and on-premise environments. -If only one deployment model is assessed, the other one should be marked as `Not assessed`. +The deployment models are assessed independently. The CaaS assessment is +complete; the on-premise assessment remains `TBD`. #### Tested quality metric and method -This result reuses the existing stack-agnostic test definition in `test.md` and adds an protoEMDS Final Stack integration-assessment perspective. +This result reuses the existing stack-agnostic test definition in `test.md` and adds a protoEMDS Final Stack integration-assessment perspective. It does not replace the historical stack-specific result files, such as `result_edc_vc.md` or `result_fiware.md`. @@ -50,39 +51,41 @@ refers to the Data Space-only catalog, specifically the internal EDC catalog and #### Assessment -Pending. +Pass. -The assessment should be completed once consolidated technical evidence is available for the protoEMDS Final Stack deployment. - -The result should clearly indicate which deployment model was assessed. It is acceptable to assess only one deployment model if evidence for the other deployment model is not available. +A new asset, policy, and contract definition were created through the +provider's Management API. A catalogue request from the consumer returned +`200 OK` and included the product with its expected metadata, DSP endpoint, +and `HttpData-PULL` and `HttpData-PUSH` distributions. #### Deployment model assessed | Deployment model | Status | Evidence | Consolidated assessment | | --- | --- | --- | --- | -| CaaS / IONOS-managed deployment | TBD | TBD | Complete this row only if evidence is collected from the IONOS-managed deployment. Otherwise mark as `Not assessed`. | -| On-premise deployment | TBD | TBD | Complete this row only if evidence is collected from an on-premise or locally managed deployment. Otherwise mark as `Not assessed`. | +| CaaS / IONOS-managed deployment | Pass | Bruno CLI output | New test product was visible from the consumer catalogue. | +| On-premise deployment | TBD | TBD | To be assessed separately. | #### Measured results | Criteria | Measured KPI | Evidence | Notes | | --- | ---: | --- | --- | -| **No Coverage:** No technical requirements are met. The solution fails to provide any functionality for the new data product in the catalog. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Minimal Coverage:** Up to 25% of the technical requirements are met. Only basic functionalities are implemented, leaving most requirements unaddressed. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Partial Coverage:** Approximately 50% of the technical requirements are met. Key functions are partially implemented, but several critical aspects are lacking. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Significant Coverage:** About 80% of the technical requirements are met. Most functionalities work as expected, with only minor gaps needing improvement. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Full Coverage:** All technical requirements are fully met. The solution provides a comprehensive, out-of-the-box solution for the new data product in the catalog. | TBD | TBD | Pending protoEMDS Final Stack assessment. | +| **No Coverage:** No technical requirements are met. The solution fails to provide any functionality for the new data product in the catalog. | Not selected | API and UI execution | Publication succeeded. | +| **Minimal Coverage:** Up to 25% of the technical requirements are met. Only basic functionalities are implemented, leaving most requirements unaddressed. | Not selected | API and UI execution | Publication succeeded without the limitations described by this level. | +| **Partial Coverage:** Approximately 50% of the technical requirements are met. Key functions are partially implemented, but several critical aspects are lacking. | Not selected | API and UI execution | The complete test path succeeded. | +| **Significant Coverage:** About 80% of the technical requirements are met. Most functionalities work as expected, with only minor gaps needing improvement. | Not selected | API and UI execution | No gap was observed in the scope of this test. | +| **Full Coverage:** All technical requirements are fully met. The solution provides a comprehensive, out-of-the-box solution for the new data product in the catalog. | **4** | API and UI execution | Asset, policy, and contract creation returned `200 OK`; the consumer catalogue returned the new product with the expected metadata and distributions. | + +**Functional Suitability Quality Metric:** 4 -**Functional Suitability Quality Metric:** TBD +The standard Management API covered the complete publication path without +custom changes. The catalogue response confirmed that the consumer could see +the new product, so this test meets the Full Coverage criterion. #### Notes -This result introduces an **protoEMDS Final Stack** perspective for the existing test `2.2.3.1A` under the KPI1 area **Catalogue publication**. +This result introduces a **protoEMDS Final Stack** perspective for the existing test `2.2.3.1A` under the KPI1 area **Catalogue publication**. This result should not be interpreted as part of the original Phase 1 / Phase 2 stack-comparison campaign. It is intended as an integration phase assessment of the current EMDS final technical infrastructure. -The assessment should be completed using consolidated technical evidence, such as endpoint responses, logs, screenshots, Postman/curl executions, GitHub issues, pull requests, repository references, deployment status or confirmation from the relevant component owner. - -This result file was generated from the local `test.md` and, where available, the local `result_edc_vc.md` structure. EDC+VC-specific evidence, values and scores were intentionally not reused. - - +The score applies to the assessed CaaS deployment. The on-premise result +remains TBD for a separate assessment. diff --git a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1b/result_protoemds_final_stack.md b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1b/result_protoemds_final_stack.md index 3912f808..2052bc6d 100644 --- a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1b/result_protoemds_final_stack.md +++ b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1b/result_protoemds_final_stack.md @@ -17,21 +17,22 @@ This section identifies the technical context in which the protoEMDS Final Stack | Existing test ID | `2.2.3.1B` | | Level | UC + Technical | | ISO/IEC 25010 mapping | Functional suitability, Compatibility | -| Owner (tentative) | Alessio (Cefriel) / Wilhelm | +| Owner | Casper (imec, @vghelu49) | | Reviewer | Alessio | -| Deployment model assessed | TBD | -| Target environment | TBD | -| EDC version / release | TBD | -| Connector deployment reference | TBD | -| Assessment evidence | TBD | +| Deployment model assessed | CaaS / IONOS-managed deployment | +| Target environment | IONOS-managed CaaS deployment | +| EDC version / release | `emds-edc-connector` `9916cc5`, based on Eclipse EDC `0.10.0` | +| Connector deployment reference | `deployEMDS-k8s-deployment`, branch `prepare-prod`, commit `846e5f1` | +| Assessment evidence | Bruno CLI API execution | The assessment should be completed using the deployment model for which evidence is realistically available. It is not mandatory to execute the same test in both CaaS and on-premise environments. -If only one deployment model is assessed, the other one should be marked as `Not assessed`. +The deployment models are assessed independently. The CaaS assessment is +complete; the on-premise assessment remains `TBD`. #### Tested quality metric and method -This result reuses the existing stack-agnostic test definition in `test.md` and adds an protoEMDS Final Stack integration-assessment perspective. +This result reuses the existing stack-agnostic test definition in `test.md` and adds a protoEMDS Final Stack integration-assessment perspective. It does not replace the historical stack-specific result files, such as `result_edc_vc.md` or `result_fiware.md`. @@ -50,39 +51,41 @@ The EMDS catalog, as defined in the relevant documentation, refers to the Data S #### Assessment -Pending. +Pass. -The assessment should be completed once consolidated technical evidence is available for the protoEMDS Final Stack deployment. - -The result should clearly indicate which deployment model was assessed. It is acceptable to assess only one deployment model if evidence for the other deployment model is not available. +The asset was created first, without a policy or contract definition, and did +not appear in the consumer catalogue. After adding the policy and contract +definition, the same asset appeared with its metadata, policy offer, and +data-service distributions. #### Deployment model assessed | Deployment model | Status | Evidence | Consolidated assessment | | --- | --- | --- | --- | -| CaaS / IONOS-managed deployment | TBD | TBD | Complete this row only if evidence is collected from the IONOS-managed deployment. Otherwise mark as `Not assessed`. | -| On-premise deployment | TBD | TBD | Complete this row only if evidence is collected from an on-premise or locally managed deployment. Otherwise mark as `Not assessed`. | +| CaaS / IONOS-managed deployment | Pass | Bruno CLI API execution | Existing asset was absent before publication and visible after its policy and contract definition were created. | +| On-premise deployment | TBD | TBD | To be assessed separately. | #### Measured results | Criteria | Measured KPI | Evidence | Notes | | --- | ---: | --- | --- | -| **No Coverage:** No technical requirements are met. The solution fails to provide any functionality for an existing data product published in the catalog. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Minimal Coverage:** Up to 25% of the technical requirements are met. Only basic functionalities are implemented, leaving most requirements unaddressed. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Partial Coverage:** Approximately 50% of the technical requirements are met. Key functions are partially implemented, but several critical aspects are lacking. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Significant Coverage:** About 80% of the technical requirements are met. Most functionalities work as expected, with only minor gaps needing improvement. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Full Coverage:** All technical requirements are fully met. The solution provides a comprehensive, out-of-the-box solution for an existing data product published in the catalog. | TBD | TBD | Pending protoEMDS Final Stack assessment. | +| **No Coverage:** No technical requirements are met. The solution fails to provide any functionality for an existing data product published in the catalog. | Not selected | API execution | Publication succeeded. | +| **Minimal Coverage:** Up to 25% of the technical requirements are met. Only basic functionalities are implemented, leaving most requirements unaddressed. | Not selected | API execution | Publication succeeded without the limitations described by this level. | +| **Partial Coverage:** Approximately 50% of the technical requirements are met. Key functions are partially implemented, but several critical aspects are lacking. | Not selected | API execution | The complete test path succeeded. | +| **Significant Coverage:** About 80% of the technical requirements are met. Most functionalities work as expected, with only minor gaps needing improvement. | Not selected | API execution | No gap was observed in the scope of this test. | +| **Full Coverage:** All technical requirements are fully met. The solution provides a comprehensive, out-of-the-box solution for an existing data product published in the catalog. | **4** | API execution | The before/after catalogue checks prove that adding the policy and contract definition published an already existing asset. | + +**Functional Suitability Quality Metric:** 4 -**Functional Suitability Quality Metric:** TBD +The before-and-after queries show that an existing asset can be published by +adding its policy and contract definition; the asset did not need to be +recreated. This meets the Full Coverage criterion. #### Notes -This result introduces an **protoEMDS Final Stack** perspective for the existing test `2.2.3.1B` under the KPI1 area **Catalogue publication**. +This result introduces a **protoEMDS Final Stack** perspective for the existing test `2.2.3.1B` under the KPI1 area **Catalogue publication**. This result should not be interpreted as part of the original Phase 1 / Phase 2 stack-comparison campaign. It is intended as an integration phase assessment of the current EMDS final technical infrastructure. -The assessment should be completed using consolidated technical evidence, such as endpoint responses, logs, screenshots, Postman/curl executions, GitHub issues, pull requests, repository references, deployment status or confirmation from the relevant component owner. - -This result file was generated from the local `test.md` and, where available, the local `result_edc_vc.md` structure. EDC+VC-specific evidence, values and scores were intentionally not reused. - - +The score applies to the assessed CaaS deployment. The on-premise result +remains TBD for a separate assessment. diff --git a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1d/result_protoemds_final_stack.md b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1d/result_protoemds_final_stack.md index 713f501d..a48b0235 100644 --- a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1d/result_protoemds_final_stack.md +++ b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_1d/result_protoemds_final_stack.md @@ -17,21 +17,22 @@ This section identifies the technical context in which the protoEMDS Final Stack | Existing test ID | `2.2.3.1D` | | Level | UC + Technical | | ISO/IEC 25010 mapping | Functional suitability, Compatibility | -| Owner (tentative) | Alessio (Cefriel) / Wilhelm | +| Owner | Casper (imec, @vghelu49) | | Reviewer | Alessio | -| Deployment model assessed | TBD | -| Target environment | TBD | -| EDC version / release | TBD | -| Connector deployment reference | TBD | -| Assessment evidence | TBD | +| Deployment model assessed | CaaS / IONOS-managed deployment | +| Target environment | IONOS-managed CaaS deployment | +| EDC version / release | `emds-edc-connector` `9916cc5`, based on Eclipse EDC `0.10.0` | +| Connector deployment reference | `deployEMDS-k8s-deployment`, branch `prepare-prod`, commit `846e5f1` | +| Assessment evidence | Bruno CLI API execution | The assessment should be completed using the deployment model for which evidence is realistically available. It is not mandatory to execute the same test in both CaaS and on-premise environments. -If only one deployment model is assessed, the other one should be marked as `Not assessed`. +The deployment models are assessed independently. The CaaS assessment is +complete; the on-premise assessment remains `TBD`. #### Tested quality metric and method -This result reuses the existing stack-agnostic test definition in `test.md` and adds an protoEMDS Final Stack integration-assessment perspective. +This result reuses the existing stack-agnostic test definition in `test.md` and adds a protoEMDS Final Stack integration-assessment perspective. It does not replace the historical stack-specific result files, such as `result_edc_vc.md` or `result_fiware.md`. @@ -49,39 +50,41 @@ The test aims to examine the process of catalog de-publication for a data produc #### Assessment -Pending. +Pass. -The assessment should be completed once consolidated technical evidence is available for the protoEMDS Final Stack deployment. - -The result should clearly indicate which deployment model was assessed. It is acceptable to assess only one deployment model if evidence for the other deployment model is not available. +The product was visible before its contract definition was deleted. The delete +request returned `204 No Content`; the next catalogue query no longer included +the product, while a direct asset lookup still returned `200 OK`. The test +policy and asset were removed afterward. #### Deployment model assessed | Deployment model | Status | Evidence | Consolidated assessment | | --- | --- | --- | --- | -| CaaS / IONOS-managed deployment | TBD | TBD | Complete this row only if evidence is collected from the IONOS-managed deployment. Otherwise mark as `Not assessed`. | -| On-premise deployment | TBD | TBD | Complete this row only if evidence is collected from an on-premise or locally managed deployment. Otherwise mark as `Not assessed`. | +| CaaS / IONOS-managed deployment | Pass | Bruno CLI API execution | Removing the contract definition removed catalogue visibility while retaining the asset. | +| On-premise deployment | TBD | TBD | To be assessed separately. | #### Measured results | **Criteria** | Measured KPI | Evidence | Notes | | --- | ---: | --- | --- | -| **No Coverage:** The solution does not provide any functionality for de-publishing a data product from the catalog. Users cannot remove or hide a data product once it is published, and achieving this requires extensive custom development or workarounds. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Minimal Coverage:** The solution meets up to 25% of the evaluation criteria. It may offer basic de-publication functionality, but this is not fully operational out of the box and requires significant technical effort or development to implement. The process is cumbersome and not intuitive for end users. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Partial Coverage:** The solution satisfies approximately 50% of the evaluation criteria. It allows for de-publishing of data products but requires some degree of customization or development to function correctly. Additionally, the process may be partially intuitive but could still pose challenges for end users in terms of usability. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Significant Coverage:** The solution covers about 80% of the evaluation criteria. It provides effective de-publication functionality with minimal development required. The de-publication process is mostly intuitive and user-friendly, with only minor usability issues or adjustments needed. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Full Coverage:** The solution fully meets all evaluation criteria. It offers complete, out-of-the-box functionality for de-publishing a data product, allowing users to easily remove or hide a data product from the catalog. The process is straightforward, intuitive, and requires no additional development or technical modifications. | TBD | TBD | Pending protoEMDS Final Stack assessment. | +| **No Coverage:** The solution does not provide any functionality for de-publishing a data product from the catalog. Users cannot remove or hide a data product once it is published, and achieving this requires extensive custom development or workarounds. | Not selected | API execution | De-publication succeeded. | +| **Minimal Coverage:** The solution meets up to 25% of the evaluation criteria. It may offer basic de-publication functionality, but this is not fully operational out of the box and requires significant technical effort or development to implement. The process is cumbersome and not intuitive for end users. | Not selected | API execution | No custom development or workaround was required. | +| **Partial Coverage:** The solution satisfies approximately 50% of the evaluation criteria. It allows for de-publishing of data products but requires some degree of customization or development to function correctly. Additionally, the process may be partially intuitive but could still pose challenges for end users in terms of usability. | Not selected | API execution | The standard Management API operation was sufficient. | +| **Significant Coverage:** The solution covers about 80% of the evaluation criteria. It provides effective de-publication functionality with minimal development required. The de-publication process is mostly intuitive and user-friendly, with only minor usability issues or adjustments needed. | Not selected | API execution | No functional gap was observed in the tested API path. | +| **Full Coverage:** The solution fully meets all evaluation criteria. It offers complete, out-of-the-box functionality for de-publishing a data product, allowing users to easily remove or hide a data product from the catalog. The process is straightforward, intuitive, and requires no additional development or technical modifications. | **4** | API execution | A standard contract-definition deletion removed the offer from the catalogue and retained the underlying asset. | + +**Functional Suitability Quality Metric:** 4 -**Functional Suitability Quality Metric:** TBD +The standard Management API removed the offer from the catalogue without +custom development or deletion of the underlying asset. This meets the Full +Coverage criterion. #### Notes -This result introduces an **protoEMDS Final Stack** perspective for the existing test `2.2.3.1D` under the KPI1 area **Catalogue publication**. +This result introduces a **protoEMDS Final Stack** perspective for the existing test `2.2.3.1D` under the KPI1 area **Catalogue publication**. This result should not be interpreted as part of the original Phase 1 / Phase 2 stack-comparison campaign. It is intended as an integration phase assessment of the current EMDS final technical infrastructure. -The assessment should be completed using consolidated technical evidence, such as endpoint responses, logs, screenshots, Postman/curl executions, GitHub issues, pull requests, repository references, deployment status or confirmation from the relevant component owner. - -This result file was generated from the local `test.md` and, where available, the local `result_edc_vc.md` structure. EDC+VC-specific evidence, values and scores were intentionally not reused. - - +The score applies to the assessed CaaS deployment. The on-premise result +remains TBD for a separate assessment. diff --git a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_3/images/publication-form-protoemds-final-stack.png b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_3/images/publication-form-protoemds-final-stack.png new file mode 100644 index 00000000..2596a406 Binary files /dev/null and b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_3/images/publication-form-protoemds-final-stack.png differ diff --git a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_3/result_protoemds_final_stack.md b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_3/result_protoemds_final_stack.md index 6728a389..fa56db8a 100644 --- a/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_3/result_protoemds_final_stack.md +++ b/tests/data_product_publication/publication/publication_on_emds_catalogue/test_2_2_3_3/result_protoemds_final_stack.md @@ -17,21 +17,22 @@ This section identifies the technical context in which the protoEMDS Final Stack | Existing test ID | `2.2.3.3` | | Level | UC + Technical | | ISO/IEC 25010 mapping | Functional suitability, Compatibility | -| Owner (tentative) | Alessio (Cefriel) / Wilhelm | +| Owner | Casper (imec, @vghelu49) | | Reviewer | Alessio | -| Deployment model assessed | TBD | -| Target environment | TBD | -| EDC version / release | TBD | -| Connector deployment reference | TBD | -| Assessment evidence | TBD | +| Deployment model assessed | CaaS / IONOS-managed deployment | +| Target environment | IONOS-managed CaaS deployment | +| EDC version / release | `emds-edc-connector` `9916cc5`, based on Eclipse EDC `0.10.0` | +| Connector deployment reference | `deployEMDS-k8s-deployment`, branch `prepare-prod`, commit `846e5f1` | +| Assessment evidence | Playwright UI verification and Bruno CLI API execution | The assessment should be completed using the deployment model for which evidence is realistically available. It is not mandatory to execute the same test in both CaaS and on-premise environments. -If only one deployment model is assessed, the other one should be marked as `Not assessed`. +The deployment models are assessed independently. The CaaS assessment is +complete; the on-premise assessment remains `TBD`. #### Tested quality metric and method -This result reuses the existing stack-agnostic test definition in `test.md` and adds an protoEMDS Final Stack integration-assessment perspective. +This result reuses the existing stack-agnostic test definition in `test.md` and adds a protoEMDS Final Stack integration-assessment perspective. It does not replace the historical stack-specific result files, such as `result_edc_vc.md` or `result_fiware.md`. @@ -49,39 +50,53 @@ The test aims to verify the availability of a GUI for publishing a data product #### Assessment -Pending. +The connector UI has separate screens for assets, policies, contract +definitions, and catalogues. The available controls include: -The assessment should be completed once consolidated technical evidence is available for the protoEMDS Final Stack deployment. +- an asset list, search box, and asset-creation form; +- core metadata, MobilityDCAT-AP, ODPS pricing, sample, and data-address tabs; +- policy creation with permission, prohibition, and obligation rules; +- contract-definition creation linking policies and one or more assets; +- a catalogue browser that queries a counterparty by DSP address and DID and + displays the resulting product and policy details. -The result should clearly indicate which deployment model was assessed. It is acceptable to assess only one deployment model if evidence for the other deployment model is not available. +After the connector password was entered, Management API requests returned +`200 OK`. The API-created product appeared in both the asset list and the +catalogue browser. A complete publication was not submitted through the UI. +The catalogue browser has no free-text search, metadata filters, or pagination +controls. + +![Asset publication form](images/publication-form-protoemds-final-stack.png) #### Deployment model assessed | Deployment model | Status | Evidence | Consolidated assessment | | --- | --- | --- | --- | -| CaaS / IONOS-managed deployment | TBD | TBD | Complete this row only if evidence is collected from the IONOS-managed deployment. Otherwise mark as `Not assessed`. | -| On-premise deployment | TBD | TBD | Complete this row only if evidence is collected from an on-premise or locally managed deployment. Otherwise mark as `Not assessed`. | +| CaaS / IONOS-managed deployment | Assessed | Playwright UI and Bruno CLI | GUI supports asset, policy, and contract-definition workflows plus basic catalogue browsing. | +| On-premise deployment | TBD | TBD | To be assessed separately. | #### Measured results | Criteria | Measured KPI | Evidence | Notes | | --- | ---: | --- | --- | -| **No Coverage:** No technical requirements are met. The solution does not have a GUI tool for publishing a data product offering into the catalogue and enabling discovery. The system does not provide any GUI means for users to interact with or manage data product offerings. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Minimal Coverage:** Up to 25% of the technical requirements are met. The solution provides limited functionality, such as a basic GUI tool with minimal features, allowing for partial publication of data products but lacks robust discovery options. User interface might be clunky, and only a few essential functions are available. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Partial Coverage:** Approximately 50% of the technical requirements are met. The solution includes a GUI tool that allows for the publication of data products and some discovery features. However, it lacks advanced functionality, such as customizable search options, detailed metadata management, or integration with other systems. Usability and user experience are moderately acceptable. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Significant Coverage:** About 80% of the technical requirements are met. The solution offers a comprehensive GUI tool with most required features, such as advanced publication workflows, robust discovery capabilities, customizable search filters, and metadata management. Integration with other systems and platforms is partially supported, and the user experience is generally smooth. However, some advanced features or full system integration might still be lacking. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Full Coverage:** All technical requirements are fully met. The solution includes a fully featured GUI tool for publishing data product offerings into the catalogue, complete with advanced discovery features, customizable search options, detailed metadata management, and full integration with other systems. The user interface is intuitive and user-friendly, providing an excellent user experience. | TBD | TBD | Pending protoEMDS Final Stack assessment. | +| **No Coverage:** No technical requirements are met. The solution does not have a GUI tool for publishing a data product offering into the catalogue and enabling discovery. The system does not provide any GUI means for users to interact with or manage data product offerings. | Not selected | UI inspection | A native connector GUI is deployed. | +| **Minimal Coverage:** Up to 25% of the technical requirements are met. The solution provides limited functionality, such as a basic GUI tool with minimal features, allowing for partial publication of data products but lacks robust discovery options. User interface might be clunky, and only a few essential functions are available. | Not selected | UI inspection | The GUI covers all three publication resources and basic discovery. | +| **Partial Coverage:** Approximately 50% of the technical requirements are met. The solution includes a GUI tool that allows for the publication of data products and some discovery features. However, it lacks advanced functionality, such as customizable search options, detailed metadata management, or integration with other systems. Usability and user experience are moderately acceptable. | **2** | UI and API execution | Publication forms and detailed metadata management are available, and catalogue browsing works. Advanced discovery filters, pagination, and external integrations were not demonstrated. | +| **Significant Coverage:** About 80% of the technical requirements are met. The solution offers a comprehensive GUI tool with most required features, such as advanced publication workflows, robust discovery capabilities, customizable search filters, and metadata management. Integration with other systems and platforms is partially supported, and the user experience is generally smooth. However, some advanced features or full system integration might still be lacking. | Not selected | UI inspection | Robust discovery and customizable search filters are absent. | +| **Full Coverage:** All technical requirements are fully met. The solution includes a fully featured GUI tool for publishing data product offerings into the catalogue, complete with advanced discovery features, customizable search options, detailed metadata management, and full integration with other systems. The user interface is intuitive and user-friendly, providing an excellent user experience. | Not selected | UI inspection | Advanced discovery and full external integration were not demonstrated. | + +**Functional Suitability Quality Metric:** 2 -**Functional Suitability Quality Metric:** TBD +The UI provides the main publication forms and basic catalogue browsing, but a +full UI submission was not executed. The missing search, filtering, and +pagination controls also limit the discovery workflow. This matches the +Partial Coverage criterion. #### Notes -This result introduces an **protoEMDS Final Stack** perspective for the existing test `2.2.3.3` under the KPI1 area **Catalogue publication**. +This result introduces a **protoEMDS Final Stack** perspective for the existing test `2.2.3.3` under the KPI1 area **Catalogue publication**. This result should not be interpreted as part of the original Phase 1 / Phase 2 stack-comparison campaign. It is intended as an integration phase assessment of the current EMDS final technical infrastructure. -The assessment should be completed using consolidated technical evidence, such as endpoint responses, logs, screenshots, Postman/curl executions, GitHub issues, pull requests, repository references, deployment status or confirmation from the relevant component owner. - -This result file was generated from the local `test.md` and, where available, the local `result_edc_vc.md` structure. EDC+VC-specific evidence, values and scores were intentionally not reused. - - +The score applies to the assessed CaaS deployment. The on-premise result +remains TBD for a separate assessment. diff --git a/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_1/images/catalogue-browser-protoemds-final-stack.png b/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_1/images/catalogue-browser-protoemds-final-stack.png new file mode 100644 index 00000000..176bca8a Binary files /dev/null and b/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_1/images/catalogue-browser-protoemds-final-stack.png differ diff --git a/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_1/result_protoemds_final_stack.md b/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_1/result_protoemds_final_stack.md index 1bada4a4..d8072994 100644 --- a/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_1/result_protoemds_final_stack.md +++ b/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_1/result_protoemds_final_stack.md @@ -17,21 +17,22 @@ This section identifies the technical context in which the protoEMDS Final Stack | Existing test ID | `3.1.1.1` | | Level | UC + Technical | | ISO/IEC 25010 mapping | Functional suitability, Compatibility | -| Owner (tentative) | Alessio (Cefriel) / Wilhelm | +| Owner | Casper (imec, @vghelu49) | | Reviewer | Carlos | -| Deployment model assessed | TBD | -| Target environment | TBD | -| EDC version / release | TBD | -| Connector deployment reference | TBD | -| Assessment evidence | TBD | +| Deployment model assessed | CaaS / IONOS-managed deployment | +| Target environment | IONOS-managed CaaS deployment | +| EDC version / release | `emds-edc-connector` `9916cc5`, based on Eclipse EDC `0.10.0` | +| Connector deployment reference | `deployEMDS-k8s-deployment`, branch `prepare-prod`, commit `846e5f1` | +| Assessment evidence | Bruno CLI API execution and Playwright UI verification | The assessment should be completed using the deployment model for which evidence is realistically available. It is not mandatory to execute the same test in both CaaS and on-premise environments. -If only one deployment model is assessed, the other one should be marked as `Not assessed`. +The deployment models are assessed independently. The CaaS assessment is +complete; the on-premise assessment remains `TBD`. #### Tested quality metric and method -This result reuses the existing stack-agnostic test definition in `test.md` and adds an protoEMDS Final Stack integration-assessment perspective. +This result reuses the existing stack-agnostic test definition in `test.md` and adds a protoEMDS Final Stack integration-assessment perspective. It does not replace the historical stack-specific result files, such as `result_edc_vc.md` or `result_fiware.md`. @@ -51,39 +52,45 @@ The criteria for evaluation include being open-source, a hosted solution, or par #### Assessment -Pending. +The catalogue was queried through both the Management API and the connector +UI. The API returned `200 OK` with the expected dataset metadata and DSP +distributions. The UI accepted a counterparty DSP address and DID, listed the +available products, and displayed product and policy details. -The assessment should be completed once consolidated technical evidence is available for the protoEMDS Final Stack deployment. +The UI has no free-text search, metadata filters, or pagination controls. No +external search platform was connected during this test. -The result should clearly indicate which deployment model was assessed. It is acceptable to assess only one deployment model if evidence for the other deployment model is not available. +![Catalogue browser showing MobilityDCAT-AP metadata and quality information](images/catalogue-browser-protoemds-final-stack.png) #### Deployment model assessed | Deployment model | Status | Evidence | Consolidated assessment | | --- | --- | --- | --- | -| CaaS / IONOS-managed deployment | TBD | TBD | Complete this row only if evidence is collected from the IONOS-managed deployment. Otherwise mark as `Not assessed`. | -| On-premise deployment | TBD | TBD | Complete this row only if evidence is collected from an on-premise or locally managed deployment. Otherwise mark as `Not assessed`. | +| CaaS / IONOS-managed deployment | Assessed | Bruno CLI and Playwright | API and UI catalogue retrieval passed; advanced search and external integration remain limitations. | +| On-premise deployment | TBD | TBD | To be assessed separately. | #### Measured results | **Criteria** | Measured KPI | Evidence | Notes | | --- | ---: | --- | --- | -| **No Coverage:** The solution fails to provide a native online user experience (U/X), exposes no search features, and does not offer any integration with open-source solutions, hosted solutions, or EU-driven projects. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Minimal Coverage:** The solution meets up to 25% of the evaluation criteria. This might include a basic online user interface with limited functionality, minimal search features, or an API that is available but requires significant technical effort to integrate with any of the three types of solutions: open-source, hosted, or EU-driven projects. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Partial Coverage:** The solution satisfies approximately 50% of the evaluation criteria. This could involve a functional online user experience with some search features and an API that supports integration with at least one of the three types of solutions (open-source, hosted, or EU-driven projects) but requires moderate effort. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Significant Coverage:** The solution covers about 80% of the evaluation criteria. This includes a well-developed online user experience with comprehensive search features, and an API that is well-documented and supports integration with two of the three types of solutions with minimal technical effort. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Full Coverage:** The solution fully meets all evaluation criteria. This includes a fully developed native online user experience with advanced search features, and an API that seamlessly integrates with all three types of solutions (open-source, hosted, and EU-driven projects) with minimal or no technical effort. | TBD | TBD | Pending protoEMDS Final Stack assessment. | +| **No Coverage:** The solution fails to provide a native online user experience (U/X), exposes no search features, and does not offer any integration with open-source solutions, hosted solutions, or EU-driven projects. | Not selected | API and UI execution | Both API and native UI catalogue retrieval are available. | +| **Minimal Coverage:** The solution meets up to 25% of the evaluation criteria. This might include a basic online user interface with limited functionality, minimal search features, or an API that is available but requires significant technical effort to integrate with any of the three types of solutions: open-source, hosted, or EU-driven projects. | Not selected | API and UI execution | The UI provides product browsing and detailed metadata, exceeding minimal coverage. | +| **Partial Coverage:** The solution satisfies approximately 50% of the evaluation criteria. This could involve a functional online user experience with some search features and an API that supports integration with at least one of the three types of solutions (open-source, hosted, or EU-driven projects) but requires moderate effort. | **2** | API and UI execution | The UI retrieves and displays catalogues and product details, while the DCAT JSON-LD API is usable by external tools. Free-text search, metadata filters, pagination, and a completed external integration were not demonstrated. | +| **Significant Coverage:** The solution covers about 80% of the evaluation criteria. This includes a well-developed online user experience with comprehensive search features, and an API that is well-documented and supports integration with two of the three types of solutions with minimal technical effort. | Not selected | UI inspection | Comprehensive search and two low-effort integrations were not demonstrated. | +| **Full Coverage:** The solution fully meets all evaluation criteria. This includes a fully developed native online user experience with advanced search features, and an API that seamlessly integrates with all three types of solutions (open-source, hosted, and EU-driven projects) with minimal or no technical effort. | Not selected | UI inspection | Advanced search and seamless integration with all three solution types were not demonstrated. | -**Functional Suitability Quality Metric:** TBD +**Functional Suitability Quality Metric:** 2 + +The deployment provides a working catalogue browser and a DCAT JSON-LD API, +but not the search features required for a higher score. Since no external +search integration was tested either, the result matches the Partial Coverage +criterion. #### Notes -This result introduces an **protoEMDS Final Stack** perspective for the existing test `3.1.1.1` under the KPI1 area **Catalogue discovery / metadata**. +This result introduces a **protoEMDS Final Stack** perspective for the existing test `3.1.1.1` under the KPI1 area **Catalogue discovery / metadata**. This result should not be interpreted as part of the original Phase 1 / Phase 2 stack-comparison campaign. It is intended as an integration phase assessment of the current EMDS final technical infrastructure. -The assessment should be completed using consolidated technical evidence, such as endpoint responses, logs, screenshots, Postman/curl executions, GitHub issues, pull requests, repository references, deployment status or confirmation from the relevant component owner. - -This result file was generated from the local `test.md` and, where available, the local `result_edc_vc.md` structure. EDC+VC-specific evidence, values and scores were intentionally not reused. - - +The score applies to the assessed CaaS deployment. The on-premise result +remains TBD for a separate assessment. diff --git a/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_4/result_protoemds_final_stack.md b/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_4/result_protoemds_final_stack.md index 116252f4..8339dbb1 100644 --- a/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_4/result_protoemds_final_stack.md +++ b/tests/data_product_survey/discover/consult_data_space_catalogue/test_3_1_1_4/result_protoemds_final_stack.md @@ -17,21 +17,22 @@ This section identifies the technical context in which the protoEMDS Final Stack | Existing test ID | `3.1.1.4` | | Level | UC + Technical | | ISO/IEC 25010 mapping | Functional suitability, Compatibility | -| Owner (tentative) | Alessio (Cefriel) / Wilhelm | +| Owner | Casper (imec, @vghelu49) | | Reviewer | Carlos | -| Deployment model assessed | TBD | -| Target environment | TBD | -| EDC version / release | TBD | -| Connector deployment reference | TBD | -| Assessment evidence | TBD | +| Deployment model assessed | CaaS / IONOS-managed deployment | +| Target environment | IONOS-managed CaaS deployment | +| EDC version / release | `emds-edc-connector` `9916cc5`, based on Eclipse EDC `0.10.0` | +| Connector deployment reference | `deployEMDS-k8s-deployment`, branch `prepare-prod`, commit `846e5f1` | +| Assessment evidence | Bruno CLI API execution and Playwright UI verification | The assessment should be completed using the deployment model for which evidence is realistically available. It is not mandatory to execute the same test in both CaaS and on-premise environments. -If only one deployment model is assessed, the other one should be marked as `Not assessed`. +The deployment models are assessed independently. The CaaS assessment is +complete; the on-premise assessment remains `TBD`. #### Tested quality metric and method -This result reuses the existing stack-agnostic test definition in `test.md` and adds an protoEMDS Final Stack integration-assessment perspective. +This result reuses the existing stack-agnostic test definition in `test.md` and adds a protoEMDS Final Stack integration-assessment perspective. It does not replace the historical stack-specific result files, such as `result_edc_vc.md` or `result_fiware.md`. @@ -49,39 +50,55 @@ The test aims to determine whether the data product specification provides the n #### Assessment -Pending. +The test product included standard DCAT metadata, a MobilityDCAT-AP mobility +theme, and a DQV quality annotation using +`dqv:hasQualityAnnotation`, `dqv:QualityAnnotation`, `oa:hasBody`, and +`oa:hasTarget`. -The assessment should be completed once consolidated technical evidence is available for the protoEMDS Final Stack deployment. +The consumer catalogue returned `200 OK` and preserved the product title, +MobilityDCAT-AP theme, and DQV quality annotation. In the native catalogue +browser, the product detail view rendered spatial coverage and mobility theme +and mapped the quality annotation into its **Quality Description** field. The +same UI provides fields for further MobilityDCAT-AP metadata such as +georeferencing method, network coverage, reference system, rights holder, +transport mode, applicable legislation, assessment result, and intended +information service. -The result should clearly indicate which deployment model was assessed. It is acceptable to assess only one deployment model if evidence for the other deployment model is not available. +The catalogue browser does not offer quality-based search or filtering. The +API preserved the annotation body and target, but the UI did not display them +as separate fields. + +![Catalogue view showing MobilityDCAT-AP and quality metadata](../test_3_1_1_1/images/catalogue-browser-protoemds-final-stack.png) #### Deployment model assessed | Deployment model | Status | Evidence | Consolidated assessment | | --- | --- | --- | --- | -| CaaS / IONOS-managed deployment | TBD | TBD | Complete this row only if evidence is collected from the IONOS-managed deployment. Otherwise mark as `Not assessed`. | -| On-premise deployment | TBD | TBD | Complete this row only if evidence is collected from an on-premise or locally managed deployment. Otherwise mark as `Not assessed`. | +| CaaS / IONOS-managed deployment | Pass with limitations | Bruno CLI and Playwright | Profile metadata is preserved and partly rendered; profile-aware search is not available. | +| On-premise deployment | TBD | TBD | To be assessed separately. | #### Measured results | **Criteria** | Measured KPI | Evidence | Notes | | --- | ---: | --- | --- | -| **No DCAT-AP Support:** The implementation does not allow any DCAT-AP profile or functionality. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Breaks with Napcore DCAT-AP:** The implementation breaks if Napcore's DCAT-AP is used to describe data products. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Ignores Extensions but Functional:** The implementation ignores the extensions of Napcore's DCAT-AP, but the system works as expected, with extended metadata retrievable as part of the distribution. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Partial Integration:** The implementation integrates Napcore's DCAT-AP profile and utilizes it for some search and listing functionalities, but with limitations. | TBD | TBD | Pending protoEMDS Final Stack assessment. | -| **Full Integration:** The implementation fully integrates Napcore's DCAT-AP profile and utilizes it effectively for search and listing functionalities. | TBD | TBD | Pending protoEMDS Final Stack assessment. | +| **No DCAT-AP Support:** The implementation does not allow any DCAT-AP profile or functionality. | Not selected | API and UI execution | MobilityDCAT-AP and DQV metadata were accepted. | +| **Breaks with Napcore DCAT-AP:** The implementation breaks if Napcore's DCAT-AP is used to describe data products. | Not selected | API and UI execution | Publication and catalogue retrieval remained functional. | +| **Ignores Extensions but Functional:** The implementation ignores the extensions of Napcore's DCAT-AP, but the system works as expected, with extended metadata retrievable as part of the distribution. | Not selected | API and UI execution | The UI actively renders several profile fields rather than only preserving them. | +| **Partial Integration:** The implementation integrates Napcore's DCAT-AP profile and utilizes it for some search and listing functionalities, but with limitations. | **3** | API and UI execution | Metadata and DQV quality information are preserved and partly rendered in product details, but profile-aware search/filtering and complete quality-annotation rendering are missing. | +| **Full Integration:** The implementation fully integrates Napcore's DCAT-AP profile and utilizes it effectively for search and listing functionalities. | Not selected | UI inspection | The assessed catalogue browser does not use profile fields for search or filtering. | + +**Functional Suitability Quality Metric:** 3 -**Functional Suitability Quality Metric:** TBD +The profile is more than pass-through metadata: the UI exposes +MobilityDCAT-AP fields and displays mobility and quality information in the +product details. Profile-aware search is missing, and not every part of the +DQV annotation is displayed, so the result matches Partial Integration. #### Notes -This result introduces an **protoEMDS Final Stack** perspective for the existing test `3.1.1.4` under the KPI1 area **Catalogue discovery / metadata**. +This result introduces a **protoEMDS Final Stack** perspective for the existing test `3.1.1.4` under the KPI1 area **Catalogue discovery / metadata**. This result should not be interpreted as part of the original Phase 1 / Phase 2 stack-comparison campaign. It is intended as an integration phase assessment of the current EMDS final technical infrastructure. -The assessment should be completed using consolidated technical evidence, such as endpoint responses, logs, screenshots, Postman/curl executions, GitHub issues, pull requests, repository references, deployment status or confirmation from the relevant component owner. - -This result file was generated from the local `test.md` and, where available, the local `result_edc_vc.md` structure. EDC+VC-specific evidence, values and scores were intentionally not reused. - - +The score applies to the assessed CaaS deployment. The on-premise result +remains TBD for a separate assessment.