Distributed CWL WRF Workflow

Distributed CWL WRF Workflow #

This tutorial runs the WRF part of the existing WRF/EMEP Linear Workflow across multiple compute nodes. The workflow is written in CWL, while its REAL and WRF tools use mpirun to distribute model processes over Karolina nodes allocated by the scheduler.

Important

This tutorial does not run WRF inside a container. It loads WRF and MPI from EESSI on Karolina and runs CWL with --no-container. This ensures that WRF uses the MPI environment provided by Karolina when it starts processes on multiple nodes. Do not use the container image linked from WorkflowHub for this tutorial.

How Distribution Works#

The selected CWL workflow has two main model steps:

  1. step3_real prepares the WRF boundary and initial-condition files.

  2. step4_wrf starts the weather model after REAL succeeds.

The upstream CWL tool definitions use baseCommand: mpirun and pass the realcores or wrfcores input with -np. CWL controls the dependency between REAL and WRF. MPI distributes each model step over the allocated nodes. LEXIS Platform submits the job and stages the final output dataset.

This workflow is distributed within one HPC system. It does not span two independent clusters.

Note

Before starting, log in to the LEXIS Platform and select a project with access to Karolina’s qcpu partition. Karolina’s compute nodes must also have outbound internet access because the job installs cwltool from PyPI at the start of each execution. No SSH access to Karolina is required. For help with prerequisites, see How-to Guide.

The example workflow is marked work-in-progress in WorkflowHub. Test it on a development or experimental allocation before using a production allocation.

Download the Workflow and Example Data#

On your computer, clone the source repository linked from WorkflowHub:

git clone https://github.com/UoMResearchIT/wrf_emep_cwl_linear_workflow.git
cd wrf_emep_cwl_linear_workflow

Download wrf_emep_UK_example_inputs.tar.gz from the example-data record. Extract it as specified by the workflow authors:

mkdir -p input_files
tar -xzf wrf_emep_UK_example_inputs.tar.gz \
    -C input_files --strip-components=1

This tutorial runs only workflows/wrf_workflow.cwl. It does not run the ERA5 download, WPS or EMEP sections of the complete workflow.

Prepare a Self-Contained CWL File#

The workflow imports its individual tool definitions from the upstream atmos-tools-library repository. Resolve those imports before uploading the input dataset so the compute job does not download CWL files at runtime.

Create a local CWL environment and pack the workflow:

python3 -m venv cwl-pack-venv
source cwl-pack-venv/bin/activate
python -m pip install cwltool==3.2.20260413085819
cwltool --validate workflows/wrf_workflow.cwl
cwltool --pack workflows/wrf_workflow.cwl \
    > workflows/wrf_workflow_packed.cwl
deactivate

--pack preserves the existing WorkflowHub workflow while placing its referenced CWL definitions in one file. It does not add any specific CWL syntax.

Check the WRF settings file:

example_workflow_configurations/wrf_real_cwl_settings.yaml#
generate_rundir: true
realcores: 2
wrfcores: 256

namelist_real:
  class: File
  path: '../input_files/namelists/namelist.input.wrf'

namelist_wrf:
  class: File
  path: '../input_files/namelists/namelist.input.wrf'

metdir:
  class: Directory
  path: '../input_files/wrf_met_input'

Karolina CPU nodes have 128 cores. Therefore, wrfcores: 256 demonstrates distribution across two nodes. Keep realcores small because REAL only prepares the initial and boundary-condition files.

The upstream documentation requires WRF filenames without colons when using Singularity. This tutorial runs the host application, but keeping nocolons = .true. in the WRF namelist also makes output dataset handling more portable.

Prepare the Input Dataset#

Create an archive containing only the packed workflow, its settings and the example data. Run this command from the repository root:

zip -r wrf-cwl-input.zip \
    workflows/wrf_workflow_packed.cwl \
    example_workflow_configurations/wrf_real_cwl_settings.yaml \
    input_files

First open Data Management/Datasets and upload wrf-cwl-input.zip. Enable unpacking and name the dataset WRF CWL Input. After unpacking, workflows, example_workflow_configurations and input_files must be at the dataset root. See Data Management for the complete upload procedure.

Prepare the Jobscript#

Create the following jobscript. LEXIS Platform supplies the scheduler directives, so do not add #SBATCH lines:

#!/usr/bin/env bash
set -euo pipefail

EXECUTION_DIR="$PWD"
INPUT_DIR="${EXECUTION_DIR}/input"
VENV_DIR="${EXECUTION_DIR}/cwl-runner-venv"
WORK_DIR="${EXECUTION_DIR}/cwl-work"
OUTPUT_DIR="${EXECUTION_DIR}/output"

module purge
source /cvmfs/software.eessi.io/versions/2025.06/init/bash
module load Python/3.12.3-GCCcore-13.3.0
module load WRF/4.6.1-foss-2024a-dmpar

python -m venv "${VENV_DIR}"
source "${VENV_DIR}/bin/activate"
python -m pip install \
    --disable-pip-version-check \
    cwltool==3.2.20260413085819

command -v real.exe
command -v wrf.exe
command -v mpirun
command -v cwltool

mkdir -p "${OUTPUT_DIR}"
mkdir -p "${WORK_DIR}/tmp"
mkdir -p "${WORK_DIR}/out"

echo "MPI hosts visible to this allocation:"
mpirun hostname | sort -u

cd "${INPUT_DIR}"

cwltool \
    --no-container \
    --relax-path-checks \
    --debug \
    --tmpdir-prefix "${WORK_DIR}/tmp/" \
    --tmp-outdir-prefix "${WORK_DIR}/out/" \
    --outdir "${OUTPUT_DIR}" \
    workflows/wrf_workflow_packed.cwl \
    example_workflow_configurations/wrf_real_cwl_settings.yaml

echo "WRF rank log count:"
find "${OUTPUT_DIR}" -name 'rsl.out.*' | wc -l

grep -R "SUCCESS EM_REAL INIT" "${OUTPUT_DIR}"/rsl.out.*
grep -R "SUCCESS COMPLETE WRF" "${OUTPUT_DIR}"/rsl.out.*

--no-container makes the CWL tools use the provider’s WRF and MPI installation. --relax-path-checks is required because standard WRF filenames can contain characters that CWL normally rejects. The two temporary directory options keep CWL’s WRF working files in the execution directory, which is visible to every node in the allocation.

The job creates a new virtual environment and downloads the pinned cwltool version during every execution. This keeps the workflow self-contained and removes the need for a cluster login, but it requires outbound internet access from the compute node. The other allocated cores remain idle during this short installation step.

Tip

For repeated executions, users with SSH access may create the virtual environment once under /scratch/project/PROJECT_ID after loading the same EESSI Python module. Replace the python -m venv and pip install commands with a source command pointing to that environment. A persistent environment avoids repeated downloads but ties the jobscript to that Karolina project directory. Do not copy a virtual environment created on another computer.

Upload the Jobscript#

Now open Data Management/Job Scripts and click Create Jobscript. Name it Distributed WRF CWL, select your project and the target system, paste the script, then complete the metadata and review steps. See Job Scripts section for the full upload procedure.

Create the HPC Jobscript Workflow#

Open Workflows, select HPC Jobscript, expand Distributed WRF CWL and create a workflow from the uploaded version.

  1. Name it Distributed WRF CWL Workflow.

  2. Select Karolina, its qcpu partition and your project’s Karolina computation resource.

  3. Enable input staging and select WRF CWL Input.**LEXIS Platform** makes the unpacked dataset available to the job under input.

  4. Enable output staging. Use Distributed WRF Output as the title and output as the source path.

  5. In advanced settings, set Max cores to 256. Karolina’s qcpu nodes have 128 cores, so this requests two nodes for the WRF step.

  6. Start with a walltime of 43200 seconds (12 hours), as stated by the WorkflowHub example. Shorten it after measuring a successful test run.

  7. Review the summary and create the workflow.

Warning

wrfcores controls mpirun -np inside CWL, while Max cores controls the scheduler allocation. Keep them equal. Requesting more MPI processes than allocated cores can overload a node or cause the job to fail.

Run and Monitor the Workflow#

Open the workflow details and create an execution named WRF distributed run 1. Review the selected cluster, partition, resource, walltime and core count, then create the execution.

In the Progress view, open the job task and select View HPC Job Logs. The initial MPI hosts visible to this allocation section should list more than one hostname. The CWL debug log should then show REAL completing before WRF starts. WRF creates one rsl.out.* and rsl.error.* pair per MPI rank, and the jobscript prints the final WRF rank log count. Hostnames are the authoritative confirmation of node placement; the file count confirms ranks but not which nodes hosted them.

Check the Results#

After a successful execution, open Distributed WRF Output. It should contain the WRF output files and rank logs collected by the CWL workflow, including names matching:

wrfout_d*
rsl.out.*
rsl.error.*