Transfer API
============

The Lexis Transfer API exposes endpoints for data upload and download operations.
Main goal is to provide a way in which user can transfer data between the iRODS
zone and their local storage.

This service contains two separate applications: Transfer API and Transfer Worker.
Transfer API handles data transfers between user and server. It is implemented
as a FastAPI application, which provides a RESTful interface for data transfer
operations. Transfer Worker is responsible for saving the data in the iRODS zone
and cleaning up uploaded data. Both are deployed in the LEXIS Platform Core.

Data Upload
-----------

The data upload is performed using the TUS protocol, which allows resumable uploads.
The API provides endpoints to initiate an upload session and upload data in chunks
to the server. After the initialisation of the upload session, the user gets a unique Request ID.

After successful upload of all chunks, Transfer API creates a request to the Transfer
Worker to save the uploaded data in the iRODS zone. The Transfer Worker then processes
the uploaded data and stores it in the iRODS zone. The part of this job is also to call
the Metadata API to create an empty dataset.

User can track the status of the transfer operation using the Request ID.


Data Download
-------------

The data download is performed using the HTTP protocol. In previous versions, data were initially
downloaded to the staging area and then transferred to the user.
However, this approach was inefficient and slow. Therefore, the new version of the
Transfer API allows direct download of data from the iRODS zone to the user.

If the user requests a download of a data object, the Transfer API streams
data directly from the iRODS. When a user requests a folder or an entire dataset,
the Transfer API first retrieves the list of contained items.
It then dynamically generates a ZIP archive using the `Zip Stream Library <https://pypi.org/project/zipstream-new/>`_,
which is streamed to the user in real time. This approach allows for efficient
and fast data download, as the data is streamed directly from the iRODS zone
to the user without the need for intermediate storage in the staging area.


Source Code
-----------
The source code and endpoint documentation for the Transfer API
is available at: https://opencode.it4i.eu/lexis-platform/data/api-v2.

Swagger documentation can be found at: https://api.lexis.tech/api/ddiapi/v2/transfer/docs