Workflow Orchestration#

The LEXIS Platform Orchestrator coordinates computational and data-management tasks across heterogeneous resources. It is based on Apache Airflow, where a workflow is represented as a Directed Acyclic Graph (DAG). Airflow provides dependency management, task scheduling, execution-state tracking, retries, and coordination of parallel tasks. The LEXIS Platform extends Airflow with platform-specific operators, hooks, sensors, APIs, and authentication mechanisms.

From the user’s perspective, the orchestration model separates the workflow description from the execution backend:

LEXIS Platform Workflow Definition
                |
                v
           Airflow DAG
                |
                v
          Airflow Tasks
                |
                v
        Execution Adapter
                |
       +--------+--------+
       |                 |
       v                 v
    HEAppE           Kubernetes
       |                 |
       v                 v
  HPC Scheduler   Cloud / Kubernetes

A workflow can combine tasks executed on different infrastructures while keeping dependencies, data flow, and execution state under a common orchestration model. The LEXIS Platform also integrates Airflow with the Distributed Data Infrastructure (DDI), allowing data staging to be coordinated with computation.

LEXIS Platform Workflow Definition#

The LEXIS Platform Workflow Definition (LWD) is the declarative workflow language used by the LEXIS Platform. It is YAML-based and allows a workflow to be specified without writing native Airflow Python code.

An LWD describes:

workflow metadata;
computational jobs;
dependencies between jobs (including data);
applications or command templates;
available computing locations;
execution requirements such as walltime, etc.;
input and output datasets;
parameters controlling individual computations.

A simplified definition has the following structure:

id: simulation-workflow
project_shortname: project-a

jobs:
  preprocessing:
    requirements:
      command_template_name: preprocess
      locations:
        - location_name: cluster-a

  simulation:
    requirements:
      command_template_name: simulate
      locations:
        - location_name: cluster-b
    data_inputs:
      - source: job://preprocessing/output
        target: input/

Dependencies can be expressed both as job dependencies and as data dependencies. An output produced by one job can become the input of another job, including when the two jobs execute on different infrastructures.

The LEXIS Platform translates the LWD into an Airflow DAG:

           LWD YAML
              |
              v
LEXIS Platform LWD Translator
              |
              v
   LEXIS Platform Operators
              |
              v
         Airflow DAG
              |
              v
          Execution

This translation separates the user-facing workflow model from the implementation details of Airflow. Users describe what should execute, where it may execute, and how data flows between jobs; the generated Airflow DAG contains the operators and dependencies necessary to perform those operations.

A workflow definition is reusable. A workflow execution represents a particular run with concrete input datasets, parameters, resource selections, and generated outputs.

Graphical Workflow Builder#

The LEXIS Platform Workflow Builder provides a graphical representation of the same workflow model. Instead of editing an LWD file directly, users compose a workflow from computational jobs and connect their inputs and outputs. The resulting workflow is exported as an LWD definition and subsequently follows the same translation and execution process.

Graphical Workflow
        |
        v
     LWD YAML
        |
        v
   Airflow DAG
        |
        v
    Execution

Each graphical job represents a computational operation with:

input datasets;
application or computation definition;
resource requirements.
scalar or atomic input parameters;
output data;

Connections between jobs represent dependencies. Output data from one job can be connected to the input of another:

+---------------+  data  +------------+  data  +----------------+
| Preprocessing | -----> | Simulation | -----> | Postprocessing |
+---------------+        +------------+        +----------------+

The builder therefore does not introduce a separate workflow language. It is a visual authoring layer over LWD. A workflow created graphically can be treated in the same way as an LWD authored manually.

For advanced users, LWD provides direct control over workflow definitions. The graphical builder is useful for composing and reviewing workflow topology, particularly for workflows involving multiple applications and data dependencies.

Airflow Integration with HPC through HEAppE#

For HPC execution, Airflow does not communicate directly with the local batch scheduler. The LEXIS Platform uses HEAppE Middleware as the abstraction layer between the orchestrator and HPC infrastructure.

The execution path is:

         Airflow Task
              |
              v
LEXIS Platform HEAppE Operator
              |
              v
      HEAppE Hook / API
              |
              v
      HEAppE Middleware
              |
              v
  Provider Project / Account
              |
              v
     HPC Batch Scheduler
              |
              v
        Compute Nodes

The LEXIS Platform Airflow Provider adds HEAppE-specific operators, hook, and sensor to Apache Airflow:

HEAppE Hook: Encapsulates REST API authentication and communication with the HEAppE Middleware endpoints.
HEAppE Operators: Resolves the business logic to prepare/submit/delete jobs via the HEAppE API.
HEAppE Job Sensor: Periodically checks job status asynchronously via HEAppE until execution completes, fails, or times out.

During execution, the LEXIS Platform resolves the job’s resource and associated computational project. HEAppE uses the corresponding provider-side execution context so that the job is submitted against the appropriate HPC project or account.

Airflow remains responsible for the workflow state, while HEAppE and the HPC scheduler control the actual HPC execution:

            Airflow                                                   HPC
           ---------                                                  ---

   Populate job requirements
               |
               v
 LEXIS Platform Submit Operator  --->  HEAppE (submit job)  --->   Scheduler
               |                                                       |
               v                                                       v
LEXIS Platform Job Status Sensor <--- HEAppE (check status) <---  Job running
               |                                                       |
               v                                                       v
      Airflow Task SUCCESS       <--- HEAppE (check status) <--- Job completed

This separation is important: Airflow schedules workflow tasks, which represents computational jobs; HEAppE manages the interaction with HPC; the local scheduler schedules compute jobs onto nodes.

Data movement is handled independently through DDI operators of the LEXIS Platform. A typical HPC job therefore logically consists of:

 Stage input
      |
      v
Submit HPC job
      |
      v
Monitor HPC job
      |
      v
 Stage output

LEXIS Platform Airflow Provider operators coordinate these steps so that downstream airflow’s tasks, which represent computational jobs, are started only when the required computation and data operations have completed.

Airflow Integration with Kubernetes#

Kubernetes resources follow the same orchestration principle but use a Kubernetes-specific Airflow provider instead of HEAppE.

The execution path is:

           Airflow Task
                |
                v
LEXIS Platform Kubernetes Operator
                |
                v
   Airflow Kubernetes Provider
                |
                v
          Kubernetes API
                |
                v
       Kubernetes Workload
                |
                v
       Container Execution

The provider acts as an adapter between the generic job and the Kubernetes execution environment. It translates the job’s execution specification into a Kubernetes workload and provides Airflow with the operations required to create the workload, monitor its state, obtain completion information, and propagate failures.

Parameters resolved before submission can include:

container or application reference;
command and arguments;
CPU and memory requirements;
GPU requirements where applicable;
runtime parameters and environment;
target Kubernetes resource or execution context;
input and output locations.

Airflow retains the workflow-level state:

           Airflow Task
                |
                v
LEXIS Platform Kubernetes Operator
                |
                v
   Airflow Kubernetes Provider
                |
                v
          Kubernetes API
                |
                v
        Container Workload
                |
                v
         Status / Result
                |
                v
        Airflow Task State

This is conceptually equivalent to the HEAppE integration, but the responsibility boundary is different:

Execution backend comparison#
Concept	HPC	Kubernetes
Airflow adapter	HEAppE operators/hooks	Kubernetes provider
Execution API	HEAppE API	Kubernetes API
Infrastructure manager	HPC batch scheduler	Kubernetes
Execution unit	HPC job	Container workload
Resource allocation	HPC computational project	Kubernetes resource context
Workflow state	Airflow	Airflow

The common Airflow abstraction allows an LWD workflow to combine both types of tasks:

                   +-->  HPC Task  -->    HEAppE    -->    HPC
Input Dataset -----+
                   +--> Cloud Task --> K8s Provider --> Kubernetes
                                           |
                                           v
                                     Result Dataset

The workflow author therefore operates primarily with jobs, applications, resources, datasets, and dependencies. Whether a particular job is executed through HEAppE or Kubernetes is determined by its application and Resource mapping.

Orchestration Abstraction#

The complete orchestration model can be summarised as:

                 AUTHORING
                    |
        +-----------+-----------+
        |                       |
        v                       v
    LWD YAML             Workflow Builder
        |                       |
        +-----------+-----------+
                    |
                    v
              LWD Definition
                    |
                    v
              LWD Translator
                    |
                    v
               Airflow DAG
                    |
        +-----------+-----------+
        |                       |
        v                       v
Compute Job Tasks           Data Tasks
        |                       |
  +-----+-----+                 |
  |           |                 |
  v           v                 v
HEAppE    Kubernetes           DDI
  |        Provider             |
  |           |                 |
  v           v                 v
 HPC      Kubernetes    Storage / Staging

The important architectural boundary is that LWD and the Workflow Builder describe the workflow, Airflow coordinates its execution, and providers/operators connect individual Airflow tasks to concrete execution or data-management services.

This allows workflow definitions to remain largely independent of infrastructure-specific APIs and enables a single workflow to coordinate heterogeneous HPC, Kubernetes, and data resources.

Workflow Orchestration

Contents