Transfer API

The Lexis Transfer API exposes endpoints for data upload and download operations. Main goal is to provide a way in which user can transfer data between the iRODS zone and their local storage.

This service contains two separate applications: Transfer API and Transfer Worker. Transfer API handles data transfers between user and server. It is implemented as a FastAPI application, which provides a RESTful interface for data transfer operations. Transfer Worker is responsible for saving the data in the iRODS zone and cleaning up uploaded data. Both are deployed in the LEXIS Platform Core.

Data Upload

The data upload is performed using the TUS protocol, which allows resumable uploads. The API provides endpoints to initiate an upload session and upload data in chunks to the server. After the initialisation of the upload session, the user gets a unique Request ID.

After successful upload of all chunks, Transfer API creates a request to the Transfer Worker to save the uploaded data in the iRODS zone. The Transfer Worker then processes the uploaded data and stores it in the iRODS zone. The part of this job is also to call the Metadata API to create an empty dataset.

User can track the status of the transfer operation using the Request ID.

Data Download

The data download is performed using the HTTP protocol. In previous versions, data were firstly downloaded to the staging area and then transferred to the user. However, this approach was inefficient and slow. Therefore, the new version of the Transfer API allows direct download of data from the iRODS zone to the user.

If the user requests a download of a data object, the Transfer API streams data directly from the iRODS. When a user requests a folder or an entire dataset, the Transfer API first retrieves the list of contained items. It then dynamically generates a ZIP archive using the Zip Stream Library, which is streamed to the user in real time. This approach allows for efficient and fast data download, as the data is streamed directly from the iRODS zone to the user without the need for intermediate storage in the staging area.

Source Code

The source code and endpoint documentation for the Transfer API is available at: https://opencode.it4i.eu/lexis-platform/data/api-v2.

Swagger documentation can be found at: https://api.lexis.tech/api/ddiapi/v2/transfer/docs