Distributed Data Infrastructure

This pages provides information about the data management services in the LEXIS Platform. The platform is handling all data in the form of datasets. A dataset can contain entire tree of files and folders, can be encrypted or compressed. Each dataset has a set of metadata values indexed in an ElasticSearch instance and stored as iRODS metadata.

The data management APIs exposed by the platform manage data upload, download, staging and transfer as well as metadata definition and modification. In this section we describe all services implementing the APIs and the way they are deployed on the locations.

Currently, iRODS zones are main system for storing data and their metadata in the platform, future integration with object storages such as S3 or Swift is planned.

Distributed Data Infrastructure in context of the LEXIS Platform Node

These services are deployed on each target site providing storage resources to the platform, depending on the level of integration needed.

• Staging, compression and encryption service
  • Source code
  • Staging worker
  • Encryption and compression workers
  • Data transfer API
• iRODS zone
  • Integration to LEXIS
  • Persistent unique identifiers (PIDs)
  • Users and projects management through iRODS-Keycloak syncing mechanism

Distributed Data Infrastructure in context of the LEXIS Platform Core

There services are operated in the LEXIS Platform Core and expose the common APIs for data and metadata management operations.

• Metadata API
  • Source code
  • Indexing
  • FAIR concept
  • User-provided metadata
• API documentation