.. _cwl-distributed-wrf: ######################################## Distributed CWL WRF Workflow :new:`flag` ######################################## This tutorial runs the WRF part of the existing `WRF/EMEP Linear Workflow `_ across multiple compute nodes. The workflow is written in CWL, while its REAL and WRF tools use ``mpirun`` to distribute model processes over Karolina nodes allocated by the scheduler. .. important:: This tutorial does not run WRF inside a container. It loads WRF and MPI from EESSI on Karolina and runs CWL with ``--no-container``. This ensures that WRF uses the MPI environment provided by Karolina when it starts processes on multiple nodes. Do not use the container image linked from WorkflowHub for this tutorial. How Distribution Works ====================== The selected CWL workflow has two main model steps: 1. ``step3_real`` prepares the WRF boundary and initial-condition files. 2. ``step4_wrf`` starts the weather model after REAL succeeds. The upstream CWL tool definitions use ``baseCommand: mpirun`` and pass the ``realcores`` or ``wrfcores`` input with ``-np``. CWL controls the dependency between REAL and WRF. MPI distributes each model step over the allocated nodes. **LEXIS Platform** submits the job and stages the final output dataset. This workflow is distributed within one HPC system. It does not span two independent clusters. .. note:: Before starting, log in to the `LEXIS Platform `_ and select a project with access to Karolina's ``qcpu`` partition. Karolina's compute nodes must also have outbound internet access because the job installs ``cwltool`` from PyPI at the start of each execution. No SSH access to Karolina is required. For help with prerequisites, see :ref:`howto_guide`. The example workflow is marked work-in-progress in WorkflowHub. Test it on a development or experimental allocation before using a production allocation. Download the Workflow and Example Data ====================================== On your computer, clone the source repository linked from WorkflowHub: .. code-block:: bash git clone https://github.com/UoMResearchIT/wrf_emep_cwl_linear_workflow.git cd wrf_emep_cwl_linear_workflow Download ``wrf_emep_UK_example_inputs.tar.gz`` from the `example-data record `_. Extract it as specified by the workflow authors: .. code-block:: bash mkdir -p input_files tar -xzf wrf_emep_UK_example_inputs.tar.gz \ -C input_files --strip-components=1 This tutorial runs only ``workflows/wrf_workflow.cwl``. It does not run the ERA5 download, WPS or EMEP sections of the complete workflow. Prepare a Self-Contained CWL File ================================= The workflow imports its individual tool definitions from the upstream ``atmos-tools-library`` repository. Resolve those imports before uploading the input dataset so the compute job does not download CWL files at runtime. Create a local CWL environment and pack the workflow: .. code-block:: bash python3 -m venv cwl-pack-venv source cwl-pack-venv/bin/activate python -m pip install cwltool==3.2.20260413085819 cwltool --validate workflows/wrf_workflow.cwl cwltool --pack workflows/wrf_workflow.cwl \ > workflows/wrf_workflow_packed.cwl deactivate ``--pack`` preserves the existing WorkflowHub workflow while placing its referenced CWL definitions in one file. It does not add any specific CWL syntax. Check the WRF settings file: .. code-block:: yaml :caption: example_workflow_configurations/wrf_real_cwl_settings.yaml generate_rundir: true realcores: 2 wrfcores: 256 namelist_real: class: File path: '../input_files/namelists/namelist.input.wrf' namelist_wrf: class: File path: '../input_files/namelists/namelist.input.wrf' metdir: class: Directory path: '../input_files/wrf_met_input' Karolina CPU nodes have 128 cores. Therefore, ``wrfcores: 256`` demonstrates distribution across two nodes. Keep ``realcores`` small because REAL only prepares the initial and boundary-condition files. The upstream documentation requires WRF filenames without colons when using Singularity. This tutorial runs the host application, but keeping ``nocolons = .true.`` in the WRF namelist also makes output dataset handling more portable. Prepare the Input Dataset ========================= Create an archive containing only the packed workflow, its settings and the example data. Run this command from the repository root: .. code-block:: bash zip -r wrf-cwl-input.zip \ workflows/wrf_workflow_packed.cwl \ example_workflow_configurations/wrf_real_cwl_settings.yaml \ input_files First open *Data Management/Datasets* and upload ``wrf-cwl-input.zip``. Enable unpacking and name the dataset ``WRF CWL Input``. After unpacking, ``workflows``, ``example_workflow_configurations`` and ``input_files`` must be at the dataset root. See :ref:`user-dataset-creation` for the complete upload procedure. Prepare the Jobscript =========================== Create the following jobscript. **LEXIS Platform** supplies the scheduler directives, so do not add ``#SBATCH`` lines: .. code-block:: bash #!/usr/bin/env bash set -euo pipefail EXECUTION_DIR="$PWD" INPUT_DIR="${EXECUTION_DIR}/input" VENV_DIR="${EXECUTION_DIR}/cwl-runner-venv" WORK_DIR="${EXECUTION_DIR}/cwl-work" OUTPUT_DIR="${EXECUTION_DIR}/output" module purge source /cvmfs/software.eessi.io/versions/2025.06/init/bash module load Python/3.12.3-GCCcore-13.3.0 module load WRF/4.6.1-foss-2024a-dmpar python -m venv "${VENV_DIR}" source "${VENV_DIR}/bin/activate" python -m pip install \ --disable-pip-version-check \ cwltool==3.2.20260413085819 command -v real.exe command -v wrf.exe command -v mpirun command -v cwltool mkdir -p "${OUTPUT_DIR}" mkdir -p "${WORK_DIR}/tmp" mkdir -p "${WORK_DIR}/out" echo "MPI hosts visible to this allocation:" mpirun hostname | sort -u cd "${INPUT_DIR}" cwltool \ --no-container \ --relax-path-checks \ --debug \ --tmpdir-prefix "${WORK_DIR}/tmp/" \ --tmp-outdir-prefix "${WORK_DIR}/out/" \ --outdir "${OUTPUT_DIR}" \ workflows/wrf_workflow_packed.cwl \ example_workflow_configurations/wrf_real_cwl_settings.yaml echo "WRF rank log count:" find "${OUTPUT_DIR}" -name 'rsl.out.*' | wc -l grep -R "SUCCESS EM_REAL INIT" "${OUTPUT_DIR}"/rsl.out.* grep -R "SUCCESS COMPLETE WRF" "${OUTPUT_DIR}"/rsl.out.* ``--no-container`` makes the CWL tools use the provider's WRF and MPI installation. ``--relax-path-checks`` is required because standard WRF filenames can contain characters that CWL normally rejects. The two temporary directory options keep CWL's WRF working files in the execution directory, which is visible to every node in the allocation. The job creates a new virtual environment and downloads the pinned ``cwltool`` version during every execution. This keeps the workflow self-contained and removes the need for a cluster login, but it requires outbound internet access from the compute node. The other allocated cores remain idle during this short installation step. .. tip:: For repeated executions, users with SSH access may create the virtual environment once under ``/scratch/project/PROJECT_ID`` after loading the same EESSI Python module. Replace the ``python -m venv`` and ``pip install`` commands with a ``source`` command pointing to that environment. A persistent environment avoids repeated downloads but ties the jobscript to that Karolina project directory. Do not copy a virtual environment created on another computer. Upload the Jobscript ==================== Now open *Data Management/Job Scripts* and click *Create Jobscript*. Name it ``Distributed WRF CWL``, select your project and the target system, paste the script, then complete the metadata and review steps. See :ref:`about-custom-hpc-jobs` for the full upload procedure. Create the HPC Jobscript Workflow ================================= Open *Workflows*, select *HPC Jobscript*, expand ``Distributed WRF CWL`` and create a workflow from the uploaded version. 1. Name it ``Distributed WRF CWL Workflow``. 2. Select Karolina, its ``qcpu`` partition and your project's Karolina computation resource. 3. Enable input staging and select ``WRF CWL Input``.**LEXIS Platform** makes the unpacked dataset available to the job under ``input``. 4. Enable output staging. Use ``Distributed WRF Output`` as the title and ``output`` as the source path. 5. In advanced settings, set *Max cores* to ``256``. Karolina's ``qcpu`` nodes have 128 cores, so this requests two nodes for the WRF step. 6. Start with a walltime of ``43200`` seconds (12 hours), as stated by the WorkflowHub example. Shorten it after measuring a successful test run. 7. Review the summary and create the workflow. .. warning:: ``wrfcores`` controls ``mpirun -np`` inside CWL, while *Max cores* controls the scheduler allocation. Keep them equal. Requesting more MPI processes than allocated cores can overload a node or cause the job to fail. Run and Monitor the Workflow ============================ Open the workflow details and create an execution named ``WRF distributed run 1``. Review the selected cluster, partition, resource, walltime and core count, then create the execution. In the *Progress* view, open the job task and select *View HPC Job Logs*. The initial ``MPI hosts visible to this allocation`` section should list more than one hostname. The CWL debug log should then show REAL completing before WRF starts. WRF creates one ``rsl.out.*`` and ``rsl.error.*`` pair per MPI rank, and the jobscript prints the final ``WRF rank log count``. Hostnames are the authoritative confirmation of node placement; the file count confirms ranks but not which nodes hosted them. Check the Results ================= After a successful execution, open ``Distributed WRF Output``. It should contain the WRF output files and rank logs collected by the CWL workflow, including names matching: .. code-block:: text wrfout_d* rsl.out.* rsl.error.*