LEXIS Platform Overview#
The LEXIS Platform provides a common interface for accessing distributed HPC, Kubernetes, and data-storage infrastructures. Its main purpose is to allow users and applications to execute computational workflows and manage their data without having to interact directly with the site-specific interfaces of individual infrastructure providers.
The LEXIS Platform integrates identity management, project management, compute allocations, data management, application execution, and workflow orchestration into a common platform model.
The main organisational entity is a LEXIS Platform Project. A project represents a collaboration or computational activity and provides the context in which users, compute allocations, datasets, applications, and workflows are managed.
Conceptually:
User
|
v
LEXIS Platform Project
|
+-- Users & Roles
|
+-- Compute Allocations
|
+-- Datasets
|
+-- Applications
|
+-- Workflows & Executions
Users authenticate to the LEXIS Platform through its OpenID Connect identity service. External identity federations can be used for authentication; in particular, users can log in using identities available through MyAccessID. The LEXIS Platform therefore does not require users to maintain a separate password for every participating computing site.
Authorisation is evaluated in the context of the LEXIS Platform Project. Project membership and assigned roles determine which platform operations a user can perform and which project resources the user can access.
Compute capacity is associated with a particular LEXIS Platform Project through concrete compute allocations supplied by infrastructure providers. An allocation represents an entitlement to consume a defined amount of computing capacity on a specific system, for example:
LEXIS Platform Project Name: climate-model
|
+-- HPC System A
| 50,000 CPU node-hours
|
+-- HPC System B
| 5,000 GPU node-hours
|
+-- Kubernetes Site C
Project-specific compute capacity
A single project can therefore use allocations from several infrastructure providers. The provider remains responsible for the actual scheduler, accounting system, allocation limits, and execution infrastructure, while the LEXIS Platform associates those allocations with the corresponding project and uses them when submitting computational jobs.
Data is represented through datasets managed by the Distributed Data Infrastructure (DDI) of the LEXIS Platform. Datasets provide a logical representation of files and metadata independently of their current physical location. DDI can stage data between managed storage and the filesystem of the infrastructure on which a computation is executed.
Computational workloads are represented as applications and workflow tasks. Workflows describe dependencies between computations and data movements and can execute tasks on different HPC or Kubernetes infrastructures within the same project.
The resulting LEXIS Platform model can be summarised as:
Federated Identity
|
v
LEXIS Platform User
|
v
LEXIS Platform Project
|
+-------------------+-------------------+
| | |
v v v
Compute Allocations Datasets Workflows
| | |
v | v
HPC / Kubernetes <--------+------------> Applications
|
v
Provider Infrastructure
The LEXIS Platform acts as the coordination and abstraction layer around these entities. Infrastructure providers retain control of their computing and storage systems, while users interact with them through a consistent project, identity, data, and workflow model.
LEXIS Platform Architecture#
The LEXIS Platform uses a service-oriented architecture that separates the user-facing platform services from the HPC, Kubernetes, and storage infrastructures operated by individual resource providers.
The architecture can be viewed as three main layers:
User interfaces provide access to the platform through the LEXIS Platform Portal, REST APIs, and client libraries.
LEXIS Platform Core manages identities, projects, authorisation, workflows, datasets, metadata, and the association of compute allocations with LEXIS Platform Projects.
Provider infrastructure contains the actual HPC clusters, Kubernetes environments, and storage systems together with the integration components required to connect them to the LEXIS Platform.
Conceptually:
+-------------------------------------------------------------+
| Users |
| |
| Portal / REST API / Clients |
+------------------------------+------------------------------+
|
v
+-------------------------------------------------------------+
| LEXIS Platform Core |
| |
| Keycloak UserOrg Workflow Orchestrator |
| Identity Projects/RBAC Apache Airflow |
| |
| DDI Metadata Expirio |
| Data Services Index Credentials/Tokens |
+-------------------+---------------------+-------------------+
| |
Compute Operations Data Operations
| |
v v
+-------------------------------------------------------------+
| Provider Integration |
| |
| HEAppE Kubernetes Adapter DDI Worker |
| HPC Job management Workload Execution Data Movement |
| |
| Optional iRODS |
| HTTP API & |
| iRODS Zone |
+--------------+---------------+---------------+--------------+
| | |
v v v
HPC Cluster Kubernetes Storage / HPC
Scheduler Cluster Filesystems
+-------------------------------------------------------------+
LEXIS Platform Core#
The LEXIS Platform Core provides the common services that are independent of a particular computing centre.
Keycloak provides OpenID Connect authentication and federates external identity providers such as MyAccessID. UserOrg maintains LEXIS Platform Projects, project memberships, roles, and the authorisation information used by other platform services.
The Workflow Orchestrator, based on Apache Airflow, executes workflows and coordinates computational jobs and data transfers. Workflow tasks can invoke HPC jobs through HEAppE or execute container workloads on Kubernetes.
The Distributed Data Infrastructure (DDI) manages datasets, metadata, data transfers, and staging operations. Datasets are represented logically in the LEXIS Platform independently of the storage system or compute filesystem on which their files are currently located.
Expirio manages credentials and external-service tokens required during workflow execution without exposing long-lived credentials directly to workflow definitions.
Interaction with Compute Resources#
Compute resources remain under the administrative control of the infrastructure provider. The LEXIS Platform does not replace the local scheduler, accounting system, or allocation mechanism.
Instead, a specific compute allocation on a provider system is associated with a LEXIS Platform Project. For example, a project can be assigned 50,000 node-hours on one HPC cluster and a separate GPU allocation on another site.
For HPC systems, computational jobs are submitted through HEAppE via the LEXIS Platform Airflow Provider:
Workflow
|
v
Airflow Orchestrator
|
v
LEXIS Platform Airflow Provider (HEAppE Operator / Hook)
|
v
HEAppE API / Middleware
|
v
Provider Allocation & Account Mapping
|
v
HPC Scheduler (e.g., Slurm / PBS)
|
v
Compute Nodes
HEAppE translates the platform-level execution request into a job understood by the local HPC environment. It also maps the authenticated LEXIS Platform user and project context to the appropriate local execution identity and provider allocation.
The provider scheduler remains responsible for queueing, placement, resource limits, and accounting of consumed node-hours or other compute units.
Kubernetes environments are accessed through an Airflow integration adapter. The same orchestration model is used, but the task is translated into a Kubernetes workload instead of an HPC batch job:
Workflow
|
v
Airflow
|
v
Kubernetes adapter
|
v
Kubernetes API
|
v
Pod/Job
Interaction with Storage Resources#
Data movement is handled independently from computational job submission.
The DDI maintains the logical representation of datasets and instructs provider-side DDI Workers to transfer data between managed storage and the filesystems required by computational jobs.
A typical HPC execution therefore follows:
DDI Dataset
|
v
DDI Worker
|
v
HPC Filesystem
|
v
Computational Job
|
v
HPC Filesystem
|
v
DDI Worker
|
v
Result Dataset
A provider can optionally operate an iRODS zone as a LEXIS Platform-managed storage endpoint. In this configuration, datasets can be retained at the provider site while their metadata is exposed through the common LEXIS Platform data services.
The DDI Worker can also connect directly to supported filesystems and storage endpoints, allowing a site to participate in workflow data movement without necessarily operating its own iRODS zone.
Project as the Integration Context#
The LEXIS Platform Project provides the common context connecting users, authorisation, compute allocations, data, and workflows.
Conceptually:
LEXIS Platform Project
|
+-- Users and RBAC permissions
|
+-- HPC allocation at Site A
|
+-- GPU allocation at Site B
|
+-- Kubernetes execution environment
|
+-- Datasets and storage locations
|
+-- Applications
|
+-- Workflows
This separation allows the LEXIS Platform to coordinate workflows across multiple independent providers while preserving the autonomy of each site. Providers continue to control their local identities, schedulers, compute allocations, filesystems, and storage systems; the LEXIS Platform supplies the common identity, project, workflow, and data-management layer used to access them.