Site Integration Components#

A new infrastructure provider connects to the LEXIS Platform through a small set of integration components rather than exposing its scheduler, filesystems, or storage services directly to platform users. The components deployed or configured for a site depend on whether the site contributes HPC compute, Kubernetes compute, managed storage, or a combination of these resources.

The integration is split into two independent paths:

execution integration – submits and monitors computational workloads;
data integration – moves datasets between LEXIS Platform storage and the execution environment.

For an HPC site, execution is normally integrated through HEAppE Middleware. Data movement is handled by a site-local DDI Worker. A site that also contributes persistent LEXIS Platform-managed storage can additionally operate an iRODS zone, and a site that wants users to reach its storage or cluster filesystem directly can additionally run a Transfer API. Only HEAppE and the DDI Worker are needed for a site to take part in workflows; the iRODS zone and the Transfer API are optional additions.

LEXIS Platform Core
    |
    +-- Orchestrator / Airflow
    |        |
    |        +-- HEAppE API -----------------> HPC Cluster
    |        |
    |        +-- Kubernetes Provider --------> Kubernetes Cluster
    |
    +-- DDI Staging API
             |
             +-- DDI Worker -----------------> HPC / Kubernetes Filesystems
                      |
                      +-----------------------> Optional iRODS HTTP API
                                                       |
                                                       v
                                                  iRODS Zone

This separation allows a site to expose compute resources without becoming a storage provider, or to expose storage independently of a particular compute system.

HEAppE for HPC Job Management#

HEAppE Middleware is the main integration component for exposing an HPC cluster to the LEXIS Platform. It provides a stable API between the LEXIS Platform Orchestrator and the site’s batch-processing environment, so Airflow does not need direct knowledge of the local scheduler or account model.

HEAppE is responsible for operations such as:

mapping LEXIS Platform execution requests to the appropriate site-side user, project, or accounting context;
exposing predefined command templates and supported applications;
submitting jobs to the local batch scheduler;
passing job parameters and resource requirements;
returning job identifiers and execution state;
providing access to job output and logs where supported;
cancelling or otherwise controlling submitted jobs.

The execution path is:

  LEXIS Platform Workflow Task
               |
               v
LEXIS Platform Airflow Provider
               |
               v
           HEAppE API
               |
               v
        Site-Side HEAppE
               |
               v
  Local Project / Account Mapping
               |
               v
        Batch Scheduler
               |
               v
         Compute Nodes

HEAppE deliberately preserves the site’s existing allocation and scheduling model. The LEXIS Platform selects a platform Resource associated with a LEXIS Platform Computational Project; the corresponding provider allocation is then used by HEAppE when submitting the job. CPU time, GPU time, queue limits, and other usage remain accounted by the local infrastructure.

A site therefore remains authoritative for:

scheduler configuration and policies;
local accounts and project allocations;
available queues or partitions;
installed software and command templates;
compute-time accounting;
job execution and node allocation.

The LEXIS Platform is authoritative for the higher-level mapping between a LEXIS Platform Project, its Computational Project, the exposed Resource, and the provider-side allocation.

For a new HPC site, HEAppE is the primary boundary that must be integrated with the site’s scheduler and identity/accounting model. The LEXIS Platform Airflow integration communicates with HEAppE through the HEAppE operators and hooks provided by the LEXIS Platform Airflow Provider.

DDI Worker for Site-local Data Movement#

The DDI Worker connects the site’s local filesystems and storage endpoints to the Distributed Data Infrastructure of the LEXIS Platform. Each connected centre operates a worker responsible for data operations that must be executed close to its infrastructure.

The central Staging API receives staging requests from workflows and dispatches them to the worker associated with the relevant site or location. Requests are distributed as Celery tasks through a central Redis broker; the worker keeps an open connection to it and listens on the queue belonging to its own location. The worker performs the actual transfer and reports progress and completion status back through the same task infrastructure.

 Airflow / LEXIS Platform Service
                |
                v
         DDI Staging API
                |
                v
        Celery Task Queue
                |
                v
         Site DDI Worker
                |
      +---------+---------+
      |                   |
      v                   v
HPC Filesystem     Storage Location

The worker can manage site-local connections such as:

HPC filesystems accessed through SSH/SFTP;
local or mounted POSIX/NFS storage;
an iRODS zone;
storage or filesystem endpoints associated with Cloud or Kubernetes resources.

The DDI Worker is independent of the job-management path. HEAppE can submit an HPC job while the DDI Worker separately prepares its input and collects its output.

A typical HPC workflow therefore uses both components:

    DDI Dataset
         |
         | DDI Worker
         v
HPC Input Directory
         |
         | HEAppE
         v
 HPC Job Execution
         |
         | DDI Worker
         v
 DDI Result Dataset

This separation is important for multi-site workflows. Data transfer can be planned according to dataset locations while execution can independently be mapped to a suitable compute Resource.

For a compute-only site, the DDI Worker is sufficient for making the cluster filesystem reachable by DDI staging; the site does not need to provide its own persistent LEXIS Platform data repository.

Optional Transfer API for Direct Data Access#

A site can additionally run a Transfer API instance. Where the DDI Worker performs transfers that the platform requests on behalf of a workflow, the Transfer API serves transfers that a user or a client starts directly, within their own request.

It gives users two things at that site:

streaming upload of dataset files into the site’s iRODS zone, and download back out, always through the iRODS HTTP API;
the Remote File Manager – listing, upload, download, directory creation and deletion on the site’s HPC filesystem over SSH/SFTP, without an interactive SSH session.

        User / Client
              |
              v
      Site Transfer API
              |
      +-------+-------+
      |               |
      v               v
iRODS HTTP API  HPC Filesystem
      |
      v
  iRODS zone

In practice the Transfer API goes hand in hand with the iRODS zone: a site that contributes managed storage normally deploys both, so users can move data in and out of that storage themselves. A compute-only site more often runs without one.

Unlike the DDI Worker, the Transfer API does not connect to the Celery broker and consumes no task queue; it only answers HTTP calls. It is therefore not required for workflow execution. A site without it still stages data through its DDI Worker and runs workflows normally; what users lose is the ability to move data in and out of that site themselves, and the Remote File Manager view of its filesystem.

Remote file access additionally depends on how the site maps LEXIS Platform identities to local accounts – it is usable at centres that map a platform user one-to-one to a local account, and not where all platform users share a single robot account.

Optional iRODS Zone for Managed Storage#

A site that contributes persistent storage to the LEXIS Platform can operate its own iRODS zone. The zone exposes local storage through the DDI data model while allowing the physical data to remain at the provider site.

An iRODS zone provides:

persistent storage for LEXIS Platform datasets;
collections and metadata associated with stored data;
access-control enforcement for users and projects;
a standard interface for transfer between iRODS locations;
federation with other iRODS zones where required;
mapping of LEXIS Platform data resources to concrete storage capacity.

The site-local arrangement is typically:

LEXIS Platform DDI
    |
    +-- Metadata / Platform Services
    |
    +-- Site DDI Worker
    |        |
    |        +------> HPC Filesystem
    |        |
    |        +------> iRODS HTTP API --> iRODS Zone
    |
    +----------------> Other LEXIS Platform Storage Locations

The iRODS zone is optional for a site that contributes only compute resources. In that case, workflow input can be staged from an iRODS zone or another storage location operated elsewhere, and results can be returned to that external storage after execution.

When a site contributes storage as a LEXIS Platform Resource, its iRODS zone becomes part of the distributed storage layer. Dataset metadata remains visible through the platform-level DDI services while the data itself can remain physically located at the provider.

The iRODS zone should therefore be understood as the site’s managed storage endpoint, while the DDI Worker is the site’s data-movement agent. They serve different roles and can be deployed independently when the integration profile requires it.

Kubernetes Integration#

A site can also expose a Kubernetes cluster as an execution resource. In this case, workflow tasks are submitted through the Kubernetes execution adapter used by the LEXIS Platform Airflow environment rather than through the HPC scheduler path.

LEXIS Workflow Task
        |
        v
     Airflow
        |
        v
Kubernetes Provider
        |
        v
   Kubernetes API
        |
        v
Pod / Job / Workload

The adapter translates the resolved workflow task into a Kubernetes workload and monitors its lifecycle. The site remains responsible for the Kubernetes-side resource model, namespaces, quotas, container runtime, scheduling, and access policies.

Data management remains separate from execution. If a Kubernetes task consumes or produces LEXIS Platform datasets, the DDI Worker or another supported DDI-access path provides the required movement between LEXIS Platform storage and storage visible to the workload.

A Kubernetes site can therefore be connected using the same conceptual split as an HPC site:

Execution:  Airflow -> Kubernetes Provider -> Kubernetes API
Data:       DDI Staging API -> DDI Worker -> Workload Storage

An iRODS zone is required only when the site also contributes persistent LEXIS Platform-managed storage.

Site Integration Profiles#

The required components depend on the resources contributed by the site.

Typical site integration profiles#
Site capability	HEAppE	DDI Worker	iRODS zone	Transfer API
HPC compute	Required	Required for workflow data staging	Optional	Optional – adds Remote File Manager access to the cluster filesystem
HPC compute + managed storage	Required	Required	Recommended / normally provided	Recommended / normally provided
Managed storage only	Not required	Required for staging operations	Required for iRODS-backed LEXIS Platform storage	Recommended / normally provided
Kubernetes compute	Not required for direct Kubernetes integration	Required when DDI staging is used	Optional	Optional
Kubernetes compute + managed storage	Not required for direct Kubernetes integration	Required	Recommended / normally provided	Recommended / normally provided

The minimal HPC integration is therefore HEAppE + DDI Worker. Adding an iRODS zone turns the site into both a compute and a managed-data provider, and such a site normally deploys a Transfer API with it so users can move data in and out of that storage themselves. For Kubernetes, the equivalent execution-side component is the Kubernetes Airflow adapter, while the DDI components remain unchanged.

From the LEXIS Platform Core perspective, these components expose three independent abstractions:

Resource       -> where computation can execute
Allocation     -> under which provider entitlement it executes
Location       -> where data can be stored or staged

Keeping these abstractions separate allows a new site to integrate incrementally and preserves local control over compute allocations, schedulers, storage systems, and security policies.

Site Integration Components

Contents