Workflow Orchestration ====================== The **LEXIS Platform Orchestrator** coordinates computational and data-management tasks across heterogeneous resources. It is based on **Apache Airflow**, where a workflow is represented as a Directed Acyclic Graph (DAG). Airflow provides dependency management, task scheduling, execution-state tracking, retries, and coordination of parallel tasks. The LEXIS Platform extends Airflow with platform-specific operators, hooks, sensors, APIs, and authentication mechanisms. From the user's perspective, the orchestration model separates the **workflow description** from the **execution backend**: .. code-block:: text LEXIS Platform Workflow Definition | v Airflow DAG | v Airflow Tasks | v Execution Adapter | +--------+--------+ | | v v HEAppE Kubernetes | | v v HPC Scheduler Cloud / Kubernetes A workflow can combine tasks executed on different infrastructures while keeping dependencies, data flow, and execution state under a common orchestration model. The LEXIS Platform also integrates Airflow with the Distributed Data Infrastructure (DDI), allowing data staging to be coordinated with computation. LEXIS Platform Workflow Definition ----------------------------------- The **LEXIS Platform Workflow Definition (LWD)** is the declarative workflow language used by the LEXIS Platform. It is YAML-based and allows a workflow to be specified without writing native Airflow Python code. An LWD describes: * workflow metadata; * computational jobs; * dependencies between jobs (including data); * applications or command templates; * available computing locations; * execution requirements such as walltime, etc.; * input and output datasets; * parameters controlling individual computations. A simplified definition has the following structure: .. code-block:: yaml id: simulation-workflow project_shortname: project-a jobs: preprocessing: requirements: command_template_name: preprocess locations: - location_name: cluster-a simulation: requirements: command_template_name: simulate locations: - location_name: cluster-b data_inputs: - source: job://preprocessing/output target: input/ Dependencies can be expressed both as **job dependencies** and as **data dependencies**. An output produced by one job can become the input of another job, including when the two jobs execute on different infrastructures. The LEXIS Platform translates the LWD into an Airflow DAG: .. code-block:: text LWD YAML | v LEXIS Platform LWD Translator | v LEXIS Platform Operators | v Airflow DAG | v Execution This translation separates the user-facing workflow model from the implementation details of Airflow. Users describe **what should execute, where it may execute, and how data flows between jobs**; the generated Airflow DAG contains the operators and dependencies necessary to perform those operations. A workflow definition is reusable. A **workflow execution** represents a particular run with concrete input datasets, parameters, resource selections, and generated outputs. Graphical Workflow Builder -------------------------- The **LEXIS Platform Workflow Builder** provides a graphical representation of the same workflow model. Instead of editing an LWD file directly, users compose a workflow from computational jobs and connect their inputs and outputs. The resulting workflow is exported as an LWD definition and subsequently follows the same translation and execution process. .. code-block:: text Graphical Workflow | v LWD YAML | v Airflow DAG | v Execution Each graphical job represents a computational operation with: * input datasets; * application or computation definition; * resource requirements. * scalar or atomic input parameters; * output data; Connections between jobs represent dependencies. Output data from one job can be connected to the input of another: .. code-block:: text +---------------+ data +------------+ data +----------------+ | Preprocessing | -----> | Simulation | -----> | Postprocessing | +---------------+ +------------+ +----------------+ The builder therefore does not introduce a separate workflow language. It is a **visual authoring layer over LWD**. A workflow created graphically can be treated in the same way as an LWD authored manually. For advanced users, LWD provides direct control over workflow definitions. The graphical builder is useful for composing and reviewing workflow topology, particularly for workflows involving multiple applications and data dependencies. Airflow Integration with HPC through HEAppE ------------------------------------------- For HPC execution, Airflow does not communicate directly with the local batch scheduler. The LEXIS Platform uses **HEAppE Middleware** as the abstraction layer between the orchestrator and HPC infrastructure. The execution path is: .. code-block:: text Airflow Task | v LEXIS Platform HEAppE Operator | v HEAppE Hook / API | v HEAppE Middleware | v Provider Project / Account | v HPC Batch Scheduler | v Compute Nodes The **LEXIS Platform Airflow Provider** adds HEAppE-specific operators, hook, and sensor to Apache Airflow: * **HEAppE Hook:** Encapsulates REST API authentication and communication with the HEAppE Middleware endpoints. * **HEAppE Operators:** Resolves the business logic to prepare/submit/delete jobs via the HEAppE API. * **HEAppE Job Sensor:** Periodically checks job status asynchronously via HEAppE until execution completes, fails, or times out. During execution, the LEXIS Platform resolves the job's resource and associated computational project. HEAppE uses the corresponding provider-side execution context so that the job is submitted against the appropriate HPC project or account. Airflow remains responsible for the workflow state, while HEAppE and the HPC scheduler control the actual HPC execution: .. code-block:: text Airflow HPC --------- --- Populate job requirements | v LEXIS Platform Submit Operator ---> HEAppE (submit job) ---> Scheduler | | v v LEXIS Platform Job Status Sensor <--- HEAppE (check status) <--- Job running | | v v Airflow Task SUCCESS <--- HEAppE (check status) <--- Job completed This separation is important: **Airflow schedules workflow tasks, which represents computational jobs; HEAppE manages the interaction with HPC; the local scheduler schedules compute jobs onto nodes**. Data movement is handled independently through DDI operators of the LEXIS Platform. A typical HPC job therefore logically consists of: .. code-block:: text Stage input | v Submit HPC job | v Monitor HPC job | v Stage output LEXIS Platform Airflow Provider operators coordinate these steps so that downstream airflow's tasks, which represent computational jobs, are started only when the required computation and data operations have completed. Airflow Integration with Kubernetes ----------------------------------- Kubernetes resources follow the same orchestration principle but use a **Kubernetes-specific Airflow provider** instead of HEAppE. The execution path is: .. code-block:: text Airflow Task | v LEXIS Platform Kubernetes Operator | v Airflow Kubernetes Provider | v Kubernetes API | v Kubernetes Workload | v Container Execution The provider acts as an adapter between the generic job and the Kubernetes execution environment. It translates the job's execution specification into a Kubernetes workload and provides Airflow with the operations required to create the workload, monitor its state, obtain completion information, and propagate failures. Parameters resolved before submission can include: * container or application reference; * command and arguments; * CPU and memory requirements; * GPU requirements where applicable; * runtime parameters and environment; * target Kubernetes resource or execution context; * input and output locations. Airflow retains the workflow-level state: .. code-block:: text Airflow Task | v LEXIS Platform Kubernetes Operator | v Airflow Kubernetes Provider | v Kubernetes API | v Container Workload | v Status / Result | v Airflow Task State This is conceptually equivalent to the HEAppE integration, but the responsibility boundary is different: .. list-table:: Execution backend comparison :header-rows: 1 :widths: 25 35 40 * - Concept - HPC - Kubernetes * - Airflow adapter - HEAppE operators/hooks - Kubernetes provider * - Execution API - HEAppE API - Kubernetes API * - Infrastructure manager - HPC batch scheduler - Kubernetes * - Execution unit - HPC job - Container workload * - Resource allocation - HPC computational project - Kubernetes resource context * - Workflow state - Airflow - Airflow The common Airflow abstraction allows an LWD workflow to combine both types of tasks: .. code-block:: text +--> HPC Task --> HEAppE --> HPC Input Dataset -----+ +--> Cloud Task --> K8s Provider --> Kubernetes | v Result Dataset The workflow author therefore operates primarily with **jobs, applications, resources, datasets, and dependencies**. Whether a particular job is executed through HEAppE or Kubernetes is determined by its application and Resource mapping. Orchestration Abstraction ------------------------- The complete orchestration model can be summarised as: .. code-block:: text AUTHORING | +-----------+-----------+ | | v v LWD YAML Workflow Builder | | +-----------+-----------+ | v LWD Definition | v LWD Translator | v Airflow DAG | +-----------+-----------+ | | v v Compute Job Tasks Data Tasks | | +-----+-----+ | | | | v v v HEAppE Kubernetes DDI | Provider | | | | v v v HPC Kubernetes Storage / Staging The important architectural boundary is that **LWD and the Workflow Builder describe the workflow, Airflow coordinates its execution, and providers/operators connect individual Airflow tasks to concrete execution or data-management services**. This allows workflow definitions to remain largely independent of infrastructure-specific APIs and enables a single workflow to coordinate heterogeneous HPC, Kubernetes, and data resources.