Multi-Cluster Workflow #

In this tutorial, you run the same job on two HPC clusters from one workflow execution. Both jobs receive the same input files and start independently, so LEXIS Platform can submit them in parallel. Each cluster writes its own result dataset, allowing you to compare the results and execution environments.

The example calculates basic statistics for a list of numbers. It uses the JobScript command template to execute a portable shell script staged with the input dataset.

Required Values#

Before starting, identify the values that you will substitute throughout this tutorial:

Collect YOUR_PROJECT_SHORTNAME, the two location names CLUSTER_A and CLUSTER_B, their computation resources RESOURCE_A and RESOURCE_B, and the CPU node types NODE_TYPE_A and NODE_TYPE_B. See Values Used in Custom Workflow YAML for where to find these identifiers. Node type names do not need to match between clusters.

Prepare the Input Dataset#

On your computer, create a folder named multi-cluster-input:

mkdir multi-cluster-input
cd multi-cluster-input

Create compare_clusters.sh inside it:

#!/usr/bin/env bash
set -euo pipefail

INPUT_FILE="values.txt"
OUTPUT_DIR="result"

test -s "${INPUT_FILE}"
mkdir -p "${OUTPUT_DIR}"

HOST_NAME="$(hostname -f 2>/dev/null || hostname)"
INPUT_SHA256="$(sha256sum "${INPUT_FILE}" | awk '{print $1}')"

{
    printf 'Cluster label: %s\n' "${CLUSTER_LABEL:-not set}"
    printf 'Compute host: %s\n' "${HOST_NAME}"
    printf 'Input SHA-256: %s\n' "${INPUT_SHA256}"
    awk '
        NF {
            count++
            sum += $1
            if (count == 1 || $1 < min) min = $1
            if (count == 1 || $1 > max) max = $1
        }
        END {
            if (count == 0) exit 1
            printf "Values: %d\n", count
            printf "Minimum: %.6f\n", min
            printf "Maximum: %.6f\n", max
            printf "Mean: %.6f\n", sum / count
        }
    ' "${INPUT_FILE}"
} | tee "${OUTPUT_DIR}/summary.txt"

The script records a cluster label, the compute hostname and a checksum of the input before calculating the statistics. Matching checksums later confirm that both clusters processed the same data.

Create values.txt beside the script and put one number on each line:

Make the script executable, then create the archive from inside the folder so that the script and data file are at the archive root:

chmod 750 compare_clusters.sh
zip ../multi-cluster-input.zip compare_clusters.sh values.txt

Upload multi-cluster-input.zip in Data Management/Datasets, enable unpacking and name the dataset Multi-Cluster Input. See Data Management for the complete upload procedure.

If both clusters stage data from the same iRODS storage, they can use the same uploaded dataset. If each cluster uses a different iRODS storage, you may need to upload the archive once to each storage. The workflow uses ddi://~ for both inputs, so select the appropriate uploaded dataset for each cluster when creating an execution. To bind either input to a specific dataset instead, see Finding a Dataset DDI URI.

Create the Workflow#

Open Workflows from the main menu, select Custom Workflow and choose the YAML code editor. See Create Custom LEXIS Platform workflow for the complete portal procedure. Enter this definition and replace every placeholder value:

id: Parallel_multi_cluster_workflow
desc: Run the same statistics job on two clusters in parallel
project_shortname: YOUR_PROJECT_SHORTNAME
jobs:

  Run_on_cluster_A:
    requirements:
      policy: preferred
      command_template_name: JobScript
      node_type_name: NODE_TYPE_A
      locations:
        - location_name: CLUSTER_A
          location_resource: RESOURCE_A
      walltime_limit: 300
      max_cores: 1
      template_parameters:
        fileName: compare_clusters.sh
      environment_variables:
        CLUSTER_LABEL: CLUSTER_A
    data_inputs:
      - source: ddi://~
        target: ./
    data_outputs:
      - source: result/
        target: ddi://~
        metadata:
          title: Multi-Cluster Result A
          access: project

  Run_on_cluster_B:
    requirements:
      policy: preferred
      command_template_name: JobScript
      node_type_name: NODE_TYPE_B
      locations:
        - location_name: CLUSTER_B
          location_resource: RESOURCE_B
      walltime_limit: 300
      max_cores: 1
      template_parameters:
        fileName: compare_clusters.sh
      environment_variables:
        CLUSTER_LABEL: CLUSTER_B
    data_inputs:
      - source: ddi://~
        target: ./
    data_outputs:
      - source: result/
        target: ddi://~
        metadata:
          title: Multi-Cluster Result B
          access: project
metadata:
  start_date: "2026-07-01T00:00:00.000Z"
  catchup: false

Review the translated workflow and click Create Workflow.

Run the Workflow#

Open the workflow details and click Create Workflow Execution. Check both computation resources and input dataset selections, then create the execution. See Execution of LEXIS Platform Workflows for the complete execution procedure.

The execution graph shows Run_on_cluster_A and Run_on_cluster_B as separate branches. Open each task to inspect its HPC job log. One job may be queued or finish before the other without preventing the other branch from running.

Check the Results#

After both jobs finish, the execution has two output datasets:

Multi-Cluster Result A contains the summary from CLUSTER_A.
Multi-Cluster Result B contains the summary from CLUSTER_B.

Open summary.txt in each dataset. For the example input, both files should report these statistics:

Values: 5
Minimum: 10.000000
Maximum: 50.000000
Mean: 32.000000

The cluster labels and compute hostnames should differ, while the input checksums and calculated statistics should match. This confirms that the same job processed the same dataset independently on both clusters.

Multi-Cluster Workflow

Contents