.. _parallel-multi-cluster-workflow: ################################## Multi-Cluster Workflow :new:`flag` ################################## In this tutorial, you run the same job on two HPC clusters from one workflow execution. Both jobs receive the same input files and start independently, so **LEXIS Platform** can submit them in parallel. Each cluster writes its own result dataset, allowing you to compare the results and execution environments. The example calculates basic statistics for a list of numbers. It uses the ``JobScript`` command template to execute a portable shell script staged with the input dataset. =============== Required Values =============== Before starting, identify the values that you will substitute throughout this tutorial: Collect ``YOUR_PROJECT_SHORTNAME``, the two location names ``CLUSTER_A`` and ``CLUSTER_B``, their computation resources ``RESOURCE_A`` and ``RESOURCE_B``, and the CPU node types ``NODE_TYPE_A`` and ``NODE_TYPE_B``. See :ref:`custom-workflow-values` for where to find these identifiers. Node type names do not need to match between clusters. ========================= Prepare the Input Dataset ========================= On your computer, create a folder named ``multi-cluster-input``: .. code-block:: bash mkdir multi-cluster-input cd multi-cluster-input Create ``compare_clusters.sh`` inside it: .. code-block:: bash #!/usr/bin/env bash set -euo pipefail INPUT_FILE="values.txt" OUTPUT_DIR="result" test -s "${INPUT_FILE}" mkdir -p "${OUTPUT_DIR}" HOST_NAME="$(hostname -f 2>/dev/null || hostname)" INPUT_SHA256="$(sha256sum "${INPUT_FILE}" | awk '{print $1}')" { printf 'Cluster label: %s\n' "${CLUSTER_LABEL:-not set}" printf 'Compute host: %s\n' "${HOST_NAME}" printf 'Input SHA-256: %s\n' "${INPUT_SHA256}" awk ' NF { count++ sum += $1 if (count == 1 || $1 < min) min = $1 if (count == 1 || $1 > max) max = $1 } END { if (count == 0) exit 1 printf "Values: %d\n", count printf "Minimum: %.6f\n", min printf "Maximum: %.6f\n", max printf "Mean: %.6f\n", sum / count } ' "${INPUT_FILE}" } | tee "${OUTPUT_DIR}/summary.txt" The script records a cluster label, the compute hostname and a checksum of the input before calculating the statistics. Matching checksums later confirm that both clusters processed the same data. Create ``values.txt`` beside the script and put one number on each line: .. code-block:: text 10 25 30 45 50 Make the script executable, then create the archive from inside the folder so that the script and data file are at the archive root: .. code-block:: bash chmod 750 compare_clusters.sh zip ../multi-cluster-input.zip compare_clusters.sh values.txt Upload ``multi-cluster-input.zip`` in *Data Management/Datasets*, enable unpacking and name the dataset ``Multi-Cluster Input``. See :ref:`user-dataset-creation` for the complete upload procedure. If both clusters stage data from the same iRODS storage, they can use the same uploaded dataset. If each cluster uses a different iRODS storage, you may need to upload the archive once to each storage. The workflow uses ``ddi://~`` for both inputs, so select the appropriate uploaded dataset for each cluster when creating an execution. To bind either input to a specific dataset instead, see :ref:`finding-dataset-ddi-uri`. =================== Create the Workflow =================== Open *Workflows* from the main menu, select *Custom Workflow* and choose the YAML code editor. See :ref:`create-custom-lexis-workflow` for the complete portal procedure. Enter this definition and replace every placeholder value: .. code-block:: yaml id: Parallel_multi_cluster_workflow desc: Run the same statistics job on two clusters in parallel project_shortname: YOUR_PROJECT_SHORTNAME jobs: Run_on_cluster_A: requirements: policy: preferred command_template_name: JobScript node_type_name: NODE_TYPE_A locations: - location_name: CLUSTER_A location_resource: RESOURCE_A walltime_limit: 300 max_cores: 1 template_parameters: fileName: compare_clusters.sh environment_variables: CLUSTER_LABEL: CLUSTER_A data_inputs: - source: ddi://~ target: ./ data_outputs: - source: result/ target: ddi://~ metadata: title: Multi-Cluster Result A access: project Run_on_cluster_B: requirements: policy: preferred command_template_name: JobScript node_type_name: NODE_TYPE_B locations: - location_name: CLUSTER_B location_resource: RESOURCE_B walltime_limit: 300 max_cores: 1 template_parameters: fileName: compare_clusters.sh environment_variables: CLUSTER_LABEL: CLUSTER_B data_inputs: - source: ddi://~ target: ./ data_outputs: - source: result/ target: ddi://~ metadata: title: Multi-Cluster Result B access: project metadata: start_date: "2026-07-01T00:00:00.000Z" catchup: false Review the translated workflow and click *Create Workflow*. ================ Run the Workflow ================ Open the workflow details and click *Create Workflow Execution*. Check both computation resources and input dataset selections, then create the execution. See :ref:`executions-of-LEXIS-Workflows` for the complete execution procedure. The execution graph shows ``Run_on_cluster_A`` and ``Run_on_cluster_B`` as separate branches. Open each task to inspect its HPC job log. One job may be queued or finish before the other without preventing the other branch from running. ================= Check the Results ================= After both jobs finish, the execution has two output datasets: * ``Multi-Cluster Result A`` contains the summary from ``CLUSTER_A``. * ``Multi-Cluster Result B`` contains the summary from ``CLUSTER_B``. Open ``summary.txt`` in each dataset. For the example input, both files should report these statistics: .. code-block:: text Values: 5 Minimum: 10.000000 Maximum: 50.000000 Mean: 32.000000 The cluster labels and compute hostnames should differ, while the input checksums and calculated statistics should match. This confirms that the same job processed the same dataset independently on both clusters.