Data Management =============== The **Distributed Data Infrastructure (DDI) of the LEXIS Platform** provides a common data management layer across the storage and computing systems connected to the LEXIS Platform. DDI separates the logical representation of a dataset from its physical storage location. Dataset metadata is indexed centrally, while the actual files remain at storage systems operated by participating resource providers. The main responsibilities of DDI are: * management of datasets and metadata; * centralised dataset discovery and search; * provider-local storage of dataset files; * upload, download, and transfer of data; * staging of input data to computational systems; * collection of workflow outputs; * project-based dataset access control. A **dataset** represents a logical collection of files and directories together with descriptive and platform metadata. APIs and workflows normally refer to datasets through their identifiers rather than directly to provider-specific filesystem paths. DDI Architecture & Dataset Model ---------------------------------- DDI follows a distributed architecture with a centralised metadata and control plane and a provider-local data plane. The main components are: * **Metadata API and OpenSearch** for centralised dataset metadata and search; * **iRODS zones** at resource providers for managed storage of dataset files; * **iRODS HTTP API** for service-oriented access to provider-local iRODS zones; * **Staging API** for requesting asynchronous data movement between platform locations; * **Transfer API** for user-driven uploads and downloads and for remote access to HPC filesystems; * **DDI Workers** -- Celery workers at resource providers performing asynchronous transfers. Conceptually:: +-------------------------------------------------------------------------+ | LEXIS Platform Core | | | | Metadata API Staging API | | | | | | v v | | OpenSearch Celery / Redis | | Metadata Index | +------------------------------------+------------------------------------+ | v | Asynchronous Requests (Celery Tasks over Redis) | +-------------------+-------------------+ | | v v +---------------------------------+ +---------------------------------+ | Provider A | | Provider B | | | | | | DDI Worker Transfer API | | DDI Worker Transfer API | | | | | | | | | | +-------+-------+ | | +-------+-------+ | | | | | | | | +-------+-------+ | | +-------+-------+ | | | | | | | | | | v v | | v v | | iRODS HTTP API HPC Filesystem | | iRODS HTTP API HPC Filesystem | | | | | | | | v | | v | | iRODS Zone | | iRODS Zone | +---------------------------------+ +---------------------------------+ This architecture allows large datasets to remain close to the infrastructure on which they are stored or processed. The LEXIS Platform maintains a global view of the datasets without requiring their files to be copied to a central platform storage system. Each provider runs its own **DDI Worker**, and a provider can additionally run its own **Transfer API**. Both reach the same site-local endpoints, but they are driven from opposite ends. The DDI Worker is a **Celery consumer**: it keeps an open connection to the platform's Redis broker and listens on the queue that belongs to its own location. Every asynchronous request leaving the Staging API therefore ends at a worker, which is where the arrows in the diagram above point. The Transfer API is an HTTP service that clients call directly. It never connects to the broker and consumes no queue, so nothing on the asynchronous path reaches it -- the two components share the site's storage endpoints, not its task flow. All iRODS access from either component goes through the **iRODS HTTP API**; neither writes into the iRODS zone by itself. The Transfer API is optional for a site -- see `Data Transfer and Remote File Access`_ below. Metadata Management ------------------- Dataset metadata is indexed in a central **OpenSearch** cluster. OpenSearch provides a global catalogue of datasets stored across all participating providers and supports operations such as: * listing and discovering datasets; * full-text and structured search; * filtering by project, owner, dataset type, or other properties; * identifying the storage location associated with a dataset. The descriptive metadata model is based on the **DataCite Metadata Schema 4.5**. DataCite provides a standardised representation suitable for persistent identification, interoperability, discovery, and FAIR-oriented publication of research outputs. Representative DataCite properties include: * ``identifier``; * ``creators``; * ``titles``; * ``publisher``; * ``publicationYear``; * ``subjects``; * ``contributors``; * ``dates``; * ``types``; * ``relatedIdentifiers``; * ``version``; * ``rightsList``; * ``descriptions``; * ``geoLocations``; * ``fundingReferences``. The LEXIS Platform complements the DataCite object with platform-specific fields required for dataset management. .. list-table:: LEXIS Platform Dataset Metadata :header-rows: 1 :widths: 30 70 * - Name - Meaning * - ``target_system`` - Name of the target system. * - ``target_resource`` - Name of the resource assigned to the location. * - ``access`` - Dataset access level: ``user``, ``project``, or ``public``. * - ``title`` - Dataset title. * - ``datacite`` - DataCite metadata object. Required for public datasets. * - ``dataset_type`` - Optional list of dataset types. * - ``version`` - Optional dataset version. * - ``provenance`` - Optional provenance information. * - ``additionalMetadata`` - Optional additional metadata represented as a dictionary of strings. Metadata associated with a dataset is maintained together with the corresponding iRODS collection and indexed in OpenSearch. OpenSearch provides the global searchable representation, while the metadata associated with the iRODS collection remains coupled to the stored dataset. The metadata flow is conceptually: .. code-block:: none Create / Update Dataset | v Metadata API | +--> Validate Metadata | +--> Create or Update iRODS Collection Metadata | +--> Update OpenSearch Index | v Searchable Dataset This allows the same metadata model to be queried consistently regardless of the provider at which the dataset files are stored. Provider-Local Storage & Access --------------------------------- Dataset files are stored at participating resource providers using **iRODS**. A provider can operate an iRODS zone connected to its local storage infrastructure while retaining administrative control over the physical data. Conceptually:: LEXIS Platform Dataset | +--> Metadata --> OpenSearch | +--> Files ----> Provider iRODS Zone | v Physical Storage An iRODS zone manages its own collections, physical storage resources, access-control rules, metadata, and storage policies. Multiple providers can therefore contribute storage independently while exposing their datasets through the same DDI interfaces. The LEXIS Platform accesses iRODS through the **iRODS HTTP API**, which exposes REST-based operations for: * collections and data objects; * listing dataset contents; * reading and writing data; * metadata management; * access-control operations. The HTTP interface avoids requiring each LEXIS Platform service to implement the native iRODS protocol. It can use OpenID Connect/JWT authentication so that storage requests are associated with identities authenticated through the LEXIS Platform identity infrastructure. Conceptually:: User / LEXIS Platform Service | | OIDC token v iRODS HTTP API | v iRODS Zone | v Physical Storage Data Transfer & Remote File Access ------------------------------------ Besides the metadata and staging services, DDI exposes a **Transfer API** for direct, user-driven data movement. The Staging API moves data between platform locations on behalf of a workflow and does so asynchronously through DDI Workers; the Transfer API instead serves interactive clients -- the LEXIS Platform Portal, client libraries, and user scripts -- that transfer the bytes themselves within their own request. Upload & Download of Dataset Files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ User data is uploaded into iRODS as a **stream**. The client sends the file content to the Transfer API, which passes it on through the **iRODS HTTP API** into the target collection as it arrives, without first assembling the whole payload in an intermediate location. Downloading works in the opposite direction: the Transfer API reads the requested files or whole dataset directories through the same interface and streams them back to the client. Every iRODS operation takes this route -- the Transfer API never writes into the iRODS zone by itself. A Transfer API instance runs at the provider, so user data is transferred at the site where it is stored rather than through a single central endpoint. It is an optional site component, but in practice it goes together with the iRODS zone: a provider that contributes managed storage normally deploys both. A provider without one still stages data through its DDI Worker, but users cannot move data in and out of that site themselves. Conceptually:: User / Client | | stream v Transfer API | v iRODS HTTP API | v iRODS Zone This is the path used when a user uploads a new dataset from a workstation or retrieves results after a workflow execution. Remote Access to HPC Filesystems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The Transfer API is also the backend of the **Remote File Manager**, which gives users access to an HPC filesystem through ordinary API calls instead of an interactive SSH session. Through it a client can: * list the contents of a directory on the target system; * upload files from the client to the HPC filesystem; * download files from the HPC filesystem to the client; * create directories and remove files and directories. Conceptually:: User / Client | v Transfer API | | SSH / SFTP v HPC Filesystem Remote file access depends on how the site maps LEXIS Platform identities to local accounts. It is available at centres that map a platform user one-to-one to a local account, and not at centres where platform users share a single robot account. See :ref:`remote_file_manager` for the user-facing description. Data Staging & Workflow Integration ------------------------------------- Computational jobs normally operate on an HPC filesystem rather than directly on data stored in iRODS. DDI therefore provides an asynchronous staging service for moving data between provider-local iRODS storage and computational filesystems. A staging operation is initiated through the central **Staging API**. The API creates a **Celery** task and places it into the queue associated with the appropriate provider. A provider-side **DDI Worker** consumes the task and performs the actual data movement. A central message broker, typically Redis, distributes tasks between the Staging API and DDI Workers. Each worker listens only on the queue for its own location, so a task is picked up by the site it is addressed to. Conceptually:: Workflow / User | v Staging API | v Celery Queue | v DDI Worker | +---------+---------+ | | v v iRODS HTTP API HPC Filesystem | v iRODS Zone Staging is asynchronous because transfers may take significantly longer than an HTTP request. Each request receives an identifier that can be used by clients or workflow operators to track its state until completion or failure. For HPC systems, the DDI Worker accesses the computational filesystem using **SSH/SFTP**. Input staging follows the path:: iRODS Zone | v iRODS HTTP API | v DDI Worker | | SSH / SFTP v HPC Filesystem | v Computational Job Output staging uses the reverse path:: Computational Job | v HPC Filesystem | | SSH / SFTP v DDI Worker | v iRODS HTTP API | v iRODS Zone | v Result Dataset This approach does not require a LEXIS Platform-specific data-transfer service to run inside the HPC cluster. From the HPC site's perspective, the DDI Worker is an authorised external client accessing the filesystem through SSH. Staging is integrated directly with workflow orchestration. Before an HPC task starts, the orchestrator can request staging of its input datasets. After the job completes, selected output files can be returned to iRODS and registered as result datasets. The complete workflow is therefore: .. code-block:: none Dataset in iRODS | Asynchronous Staging | v iRODS HTTP API | v DDI Worker | SSH / SFTP | v HPC Filesystem | v HEAppE Job | v HPC Filesystem | SSH / SFTP | v DDI Worker | v iRODS HTTP API | v Result Dataset in iRODS | v Metadata / OpenSearch The responsibilities remain separate: **HEAppE** submits and monitors the HPC job, while **DDI** manages the datasets and the DDI Worker performs the physical movement of data. Czech Core Metadata Model (CCMM) Mapping ---------------------------------------- The LEXIS Platform uses DataCite 4.5 as the standard descriptive metadata layer. This provides a basis for mapping LEXIS Platform datasets to the **Czech Core Metadata Model (CCMM)** without introducing a separate metadata representation specifically for interoperability. DataCite does not contain a dedicated workflow-provenance model comparable to workflow-oriented dataset systems. Provenance concepts are therefore derived primarily from ``relatedIdentifiers``, ``descriptions``, ``contributors``, ``dates``, ``fundingReferences``, ``rightsList``, ``geoLocations``, ``publisher``, and related DataCite elements. DataCite to CCMM Mapping ~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: DataCite Provenance to CCMM Mapping :header-rows: 1 :widths: 18 24 24 24 10 * - DataCite provenance concept - DataCite source field(s) - Description - CCMM Dataset concept - Coverage * - lineage relation - ``relatedIdentifiers`` + ``relationType=IsDerivedFrom`` - Resource was derived from another resource. - Dataset → has related resource → RelatedResource - Good match * - source relation - ``relatedIdentifiers`` + ``relationType=IsSourceOf`` - Resource acts as the source of another resource. - Dataset → has related resource → RelatedResource - Good match * - methodological provenance - ``descriptions`` + ``descriptionType=Methods`` - Methods used to create the resource. - Dataset → has description → MethodsDescription - Good match * - technical provenance - ``descriptions`` + ``descriptionType=TechnicalInfo`` - Technical information associated with resource creation. - Dataset → has description → TechnicalDescription - Partial * - provenance agent - ``contributors`` - Agent participating in creation, curation, or publication. - Dataset → has qualified relation → Agent - Good match * - provenance role - ``contributorType`` - Role of an agent in the provenance chain. - Qualified relation → Role - Good match * - creation timestamp - ``dates`` + ``dateType=Created`` - Resource creation date. - Dataset → has time reference → CreatedDate - Good match * - update timestamp - ``dates`` + ``dateType=Updated`` - Resource update date. - Dataset → has time reference → UpdatedDate - Good match * - version lineage - ``version`` - Version information. - Dataset → has version → Version - Good match * - funding provenance - ``fundingReferences`` - Funding and grant information. - Dataset → has funding reference → FundingReference - Good match * - provenance licence - ``rightsList`` - Licence and rights information. - Dataset → has terms of use → LicenseDocument - Good match * - spatial provenance - ``geoLocations`` - Spatial origin or coverage of the resource. - Dataset → has spatial coverage → Location - Partial * - repository provenance - ``publisher`` - Publishing repository or institution. - Dataset → published by → Repository - Good match Coverage & Limitations ~~~~~~~~~~~~~~~~~~~~~~~~ The ``Coverage`` value expresses how closely the DataCite concept maps to CCMM: * **Good match** — the concept has a direct or nearly direct CCMM equivalent and can be transferred with little or no semantic loss. * **Partial** — the concept can be represented, but some semantics, structure, granularity, or context may be lost. * **Missing** — no suitable CCMM representation exists and an extension or complementary provenance model would be required. No Missing concepts were identified in the mapping above. Good matches include version information, contributors and their roles, funding information, lifecycle dates, methodological descriptions, and relationships expressed through ``relatedIdentifiers``. Partial mappings occur primarily where DataCite represents information with less structure or granularity, such as technical descriptions and spatial provenance. The mapping should be treated as a **semantic interoperability mapping**, not as a complete implementation crosswalk. Repository-specific transformation rules may still be required for identifier normalisation, controlled vocabularies, contributor and relation types, and other implementation details. For provenance information that cannot be expressed adequately through DataCite and CCMM alone, complementary standards such as **PROV-O** or **RO-Crate** can be used to provide a more detailed workflow and execution provenance model.