Statistical Service Implementation



72390-33135500European Commission – DG ProgrammeCross-cutting project on sharing statistical SERVices TITLE \* FirstCap \* MERGEFORMAT Statistical Service ImplementationSTRUVAL Structural Validation ServicePhase 1 – based on SDMX Converter 389255166232This document supports describing (statistical) services in a standardized way based on user needs expressed by Eurostat production units.00This document supports describing (statistical) services in a standardized way based on user needs expressed by Eurostat production units.13544554796155This document supports business analysts to describe statistical services in a standardized way based on user needs expressed by Eurostat production units.00This document supports business analysts to describe statistical services in a standardized way based on user needs expressed by Eurostat production units.Version 0.95 SAVEDATE \@ "dd/MM/yyyy" \* MERGEFORMAT 19/10/2015Table of Contents TOC \o "1-4" \h \z \u 1General information PAGEREF _Toc433030319 \h 41.1Service name PAGEREF _Toc433030320 \h 41.2Service version PAGEREF _Toc433030321 \h 41.3Relation to Service Definition and Specification PAGEREF _Toc433030322 \h 42Invocation Protocols PAGEREF _Toc433030323 \h 53Data-by-reference Protocols PAGEREF _Toc433030324 \h 54Canonical data models PAGEREF _Toc433030325 \h 55Non-canonical data models PAGEREF _Toc433030326 \h 56Distribution PAGEREF _Toc433030327 \h 57Service Contract PAGEREF _Toc433030328 \h 57.1Operation <validate> PAGEREF _Toc433030329 \h 57.1.1Function PAGEREF _Toc433030330 \h 57.1.2Statistical methods PAGEREF _Toc433030331 \h 57.1.3Invocation protocols PAGEREF _Toc433030332 \h 67.1.4Inputs PAGEREF _Toc433030333 \h 67.1.5Outputs PAGEREF _Toc433030334 \h 67.1.6Pre-conditions PAGEREF _Toc433030335 \h 67.1.7Post-conditions PAGEREF _Toc433030336 \h 77.1.8Metrics PAGEREF _Toc433030337 \h 77.1.9Business Exceptions PAGEREF _Toc433030338 \h 77.1.10Compensation PAGEREF _Toc433030339 \h 77.1.11Specific requirements PAGEREF _Toc433030340 \h 78Parameterization PAGEREF _Toc433030341 \h 79Requirements for security PAGEREF _Toc433030342 \h 79.1Security mechanisms PAGEREF _Toc433030343 \h 79.1.1Non-repudiation PAGEREF _Toc433030344 \h 79.1.2Integrity PAGEREF _Toc433030345 \h 79.1.3Authentication and trust domains PAGEREF _Toc433030346 \h 79.1.4Self-registration PAGEREF _Toc433030347 \h 79.1.5Authorization PAGEREF _Toc433030348 \h 89.1.6Encryption PAGEREF _Toc433030349 \h 89.1.7Data at rest PAGEREF _Toc433030350 \h 89.1.8Data in transfer (end-to-end) PAGEREF _Toc433030351 \h 89.2Data protection PAGEREF _Toc433030352 \h 810Policies PAGEREF _Toc433030353 \h 810.1Security assertions PAGEREF _Toc433030354 \h 810.2Quality of service assertions PAGEREF _Toc433030355 \h 810.3Message format assertions (compliance) PAGEREF _Toc433030356 \h 810.4Other Policies PAGEREF _Toc433030357 \h 810.5Terms of use PAGEREF _Toc433030358 \h 811Non-functional characteristics (QoS) PAGEREF _Toc433030359 \h 811.1Reliability PAGEREF _Toc433030360 \h 811.2Availability PAGEREF _Toc433030361 \h 911.3Performance PAGEREF _Toc433030362 \h 911.4Multilingual support PAGEREF _Toc433030363 \h 911.5Error handling PAGEREF _Toc433030364 \h 911.6Process metrics PAGEREF _Toc433030365 \h 912Technical Dependencies PAGEREF _Toc433030366 \h 913SOA Layering PAGEREF _Toc433030367 \h 10General informationService nameSTRUVAL Structural Validation Service version1.0Relation to Service Definition and SpecificationIn this document we describe the implementation of the Phase 1 SDMX Converter-based STRUVAL Service, along the lines set out in the service definition and specification.The STRUVAL Service is based on an extended version 5.1 of the SDMX Converter Web Service, which is a part of the SDMX Converter toolchain (alongside with the API, GUI, and command line version).The structural validations covered by this initial release of the STRUVAL Service include:Verifying that the SDMX-ML message (the dataset) is a well-formed XML document.Verifying that the structural elements in the SDMX-ML message (header, dataset, groups, series, observations, etc.) are correctly ordered and nested.Detecting misplaced, undefined, and missing dimensions and attributes at the dataset, group, series, and observation levels.Detecting invalid data format and invalid values for time-period concepts.Detecting invalid codes, based on the code lists and the dataflow constraints.Detecting duplicated observations.The result of the STRUVAL Service is a machine-readable validation report containing the overall success indication and, in case of validation errors, a list of detected errors (up to a user-configurable limit). Each detected error is characterized using a standard error code, a descriptive text, and either a line/column in the input file of the incorrect XML syntactic element, or the value dimensions for the data unit (series, observation) where the error has been detected.The initial STRUVAL Service release has the following limitations:The DSD has to be sent (i.e., embedded) with the service call, and needs to be self-contained, i.e., needs to include all code lists and other artifacts referenced from the DSD. In future releases, it will be possible to refer to the DSD, code lists, and other artifacts, via SDMX registry.Currently only a subset of SDMX-ML formats is supported. The supported formats are SDMX v2.0 Comact and SDMX v2.1 Structure-specific messages. The support for other SDMX-ML formats will be added in future releases.XML syntax errors in the SDMX-ML input messages (datasets), which cause a message not to be a well-formed XML document, are currently non-recoverable, in the sense that the validation process stops upon the first encounter of such an error.Invocation ProtocolsSTRUVAL is being developed by Eurostat in order to assist the Member States and Eurostat in the process of structural validation of statistical data file. These structure and dictionaries are defined in a DSD (dataset definition) stored in the Euro SDMX Registry. The STRUVAL Service is implemented as a SOAP/HTTP Web service that extends the existing SDMX Converter Web service interface by introducing a new service operation named "validate" which accepts:The input SDMX-ML data set, embedded in the service request.The data structure file, embedded in the service request; this is normally a dataflow with the embedded DSD and, optionally, dataflow constraints.The user-defined maximal number of validation errors to be detected and reported by STRUVAL.The STRUVAL Service returns the validation report.Data-by-reference ProtocolsIn the initial release of the STRUVAL Service, all data are passed by value. In future releases, it will be possible to refer to DSDs, dataflows, code lists, etc. stored in SDMX Registries.Canonical data modelsFiles to be successfully structurally validated must have SDMX-ML file format, and be compliant with the SDMX-ML information model.Non-canonical data modelsN/ADistributionThe Phase 1 release will be distributed within Eurostat. Later versions of STRUVAL are planned to be made available for Member States.Service ContractOperation <validate>FunctionTo validate that the given input is a valid SDMX-ML message (dataset) that conforms to the structural and coding rules defined by the SDMX standard and the given DSD/dataflow.Statistical methodsNo statistical method.Invocation protocolsSOAP/HTTPInputsParameter nameTypeDescriptioninputDatabase64BinaryThe embedded input SDMX-ML document (the dataset).dsdStructureBase64BinaryThe embedded data structure file (DSD/dataflow with constraints).maxErrorNumberintThe maximal number of validation errors to report.OutputsParameter nameTypeDescriptionreturnCodeintThe overall return code:<0 for structural errors in DSD and/or malformed input data XML document0 if no structural validation errors have been found>0 if one or more structural validation errors have bene founderrorsFoundintThe number of structural validation errors that were found, if returnCode<>0.moreErrorsBooleanSet to true if there were more errors than reported (i.e., over the user-defined error limit)ErrorsXML [0..*]Description of each encountered error, if returnCode<>0, up to the user-configured error limit.Each error has:Descriptive errorClass.Numeric code and textual description.Boolean fatal indicating that this error has forced further validation to stop.Attachment level (dataset, series, etc.) described by attachedTo string field.Either:Numeric line and column indicating the location in inputData, orSet dimensions giving the coordinates of the offending data.Pre-conditionsPre-conditionDescriptionSelf-contained data structureThe dsdStructure data structure file has to be self-contained, i.e. needs to contain all necessary structural elements: DSD, code lists, constraints.Post-conditionsPre-conditionDescriptionValidation performedreturnCode >= -1MetricsMetricKPIDescriptionDuration of processingKPI-1Duration of the structural validation processTime before processingKPI-2What is the maximum delay between service launch and start of the datafile validation?Concurrent accessKPI-3What is the maximum number of concurrent access? Maximum processing capacityKPI-4Maximum processing capacity of the service (number of file x size of file processed at the same time or during a defined period)Business ExceptionsNone. The validation result is always pensationNone. Specific requirementsThe input encoding for the embedded files in inputData and dsdStructure must be UTF-8.ParameterizationCurrently the only parameter is the maximum number of validation errors to report.Requirements for securitySecurity mechanismsNon-repudiation N/AIntegrityN/AAuthentication and trust domainsAuthentication is done using ECAS.Self-registrationDone using ECAS.AuthorizationThe initial release of the STRUVAL service does not access any external resources, and therefore does not need user authorization of that kind. However, in future releases, the service may require additional authorizations to access data stored in the registry etc.EncryptionThe STRUVAL service receives and returns plain-text data.Data at restN/AData in transfer (end-to-end)Case of confidential data file:Transmission has to be encrypted (TLS)Data protectionThe STRUVAL Service does not store any data in the file system or PoliciesSecurity assertionsUse of HTTPS as confidential data file may be sent for structure validation.Quality of service assertionsTo be elaborated based on the exploitation data:Measurement of the request processing duration.Defining the delay after which the service is stopped.Message format assertions (compliance)Message format is SDMX-ML compliant to SDMX 2.0 / 2.1.Other PoliciesNone.Terms of useGeneral term of use defined for service at Eurostat.Service security policy of Eurostat.The first version will be a Proof of Concept and not yet ready for production use.Non-functional characteristics (QoS)ReliabilityMessage returns to the user is reliable in a sense of guaranteed delivery.AvailabilityThe service should be available at minimum 95% of time during working hours 8:00 – 18:00 (only working days). This especially applies to the peak times between 10:30 and 16:00.Performance(To be defined based on the exploitation data.)Multilingual supportNo multilingual requirement here.Error handlingThe STRUVAL Service should never fail and return a SOAP Service Fault message, unless under abnormal conditions of the execution environment (a network, servlet container, Web application server, Java Virtual Machine, or operating system failure).The STRUVAL Service should always return a response described in Section REF _Ref433029093 \r \h 7.1.5 within a finite amount of time.Process metricsNumber of concurrent processingMaximum size of the file to validateMaximum duration of a validation processMaximum delay between 2 processes.Technical DependenciesThe first version of the STRUVAL Service is self-contained, and has no external technical dependencies.SOA Layering ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download