Two-Step CPU-GPU Workflow #

In this tutorial, you create a reusable two-step workflow. A CPU job converts temperatures from Celsius to Fahrenheit, then a GPU job calculates their average and saves the results to iRODS. Set up the cluster scripts once, then run the workflow repeatedly from the LEXIS Platform with a different input dataset for each execution.

Both jobs use GenericCommandTemplate to run scripts stored on the shared project filesystem.

Note

You need a LEXIS Platform project with CPU and GPU resources, SSH access to the selected HPC system and permission to write to its shared project directory.

Required Values#

Before starting, identify the values that you will substitute throughout this tutorial:

YOUR_PROJECT_SHORTNAME and YOUR_COMPUTATION_RESOURCE: Use the project and computation resource identifiers described in Values Used in Custom Workflow YAML. If separate resources provide the CPU and GPU node types, use the appropriate resource name in each YAML job.
YOUR_PROJECT_DIR: The full path to a shared project directory on the selected HPC system. On Karolina, for example, use /scratch/project/PROJECT_ID. Store the workflow scripts and shared intermediate files in this directory. Do not use your home directory: workflow jobs are launched by a service account that can access LEXIS Platform project directories, but not your personal home directory.

Prepare the Cluster Scripts#

On your local computer, create a folder named cpu-gpu-workflow for the four script files:

mkdir cpu-gpu-workflow
cd cpu-gpu-workflow

The module commands below use the module names available on Karolina. When using another HPC system, replace them with the commands that provide Python and CuPy on that system.

First create a wrapper that loads Python before starting the CPU program:

#!/usr/bin/env bash
set -e

# Load Python
module load Python

# Run the CPU program
python3 YOUR_PROJECT_DIR/cpu-gpu-workflow/cpu_prepare.py

# Copy its output to the shared project directory for the GPU job
mkdir -p YOUR_PROJECT_DIR/cpu-gpu-workflow/intermediate
cp intermediate/temperatures.txt YOUR_PROJECT_DIR/cpu-gpu-workflow/intermediate/

Save it as cpu_prepare.sh in the local cpu-gpu-workflow folder.

The CPU script converts every temperature from Celsius to Fahrenheit and writes the result to the job’s intermediate directory. The wrapper then copies the same result to the shared project directory for the GPU job:

#!/usr/bin/env python3

from pathlib import Path

celsius = [
    float(line)
    for line in Path("temperatures.txt").read_text().splitlines()
]
fahrenheit = [temperature * 1.8 + 32 for temperature in celsius]
output = "\n".join(str(temperature) for temperature in fahrenheit) + "\n"

output_dir = Path("intermediate")
output_dir.mkdir(exist_ok=True)
(output_dir / "temperatures.txt").write_text(output)

print(f"Converted {len(celsius)} temperatures")

Save it as cpu_prepare.py in the same local folder.

The GPU wrapper loads CuPy, copies the intermediate file into the job’s working directory and starts the GPU program:

#!/usr/bin/env bash
set -e

# Load CuPy
module load CuPy

# Copy the CPU output into this job's working directory
cp YOUR_PROJECT_DIR/cpu-gpu-workflow/intermediate/temperatures.txt .

# Run the GPU program
python3 YOUR_PROJECT_DIR/cpu-gpu-workflow/gpu_reduce.py

Save it as gpu_reduce.sh. The wrapper runs this CuPy program:

#!/usr/bin/env python3

from pathlib import Path

import cupy as cp


temperatures = cp.loadtxt("temperatures.txt")
average = float(cp.mean(temperatures).get())

output_dir = Path("result")
output_dir.mkdir(exist_ok=True)
result = f"Average temperature: {average:.3f} F\n"
(output_dir / "average.txt").write_text(result)

print(result, end="")

Save it as gpu_reduce.py in the same local folder. The folder should now contain:

cpu-gpu-workflow/
|-- cpu_prepare.py
|-- cpu_prepare.sh
|-- gpu_reduce.py
`-- gpu_reduce.sh

Return to the directory containing cpu-gpu-workflow and upload the whole folder. Replace USERNAME with your cluster username and YOUR_PROJECT_DIR with the full shared project directory. The following example uses Karolina; use the login address of your selected HPC system when running elsewhere:

cd ..
scp -r cpu-gpu-workflow USERNAME@karolina.it4i.cz:YOUR_PROJECT_DIR/

Finally, log in to the system and make the uploaded scripts executable:

ssh USERNAME@karolina.it4i.cz
cd YOUR_PROJECT_DIR/cpu-gpu-workflow/
chmod 750 *.sh *.py

Prepare the Input Dataset#

Create a folder named cpu-gpu-input containing temperatures.txt. Enter one temperature in degrees Celsius per line:

Create the archive from inside the folder so that it does not add an extra directory level:

cd cpu-gpu-input
zip ../cpu-gpu-input.zip temperatures.txt

Upload cpu-gpu-input.zip in Data Management/Datasets and enable unpacking, so that temperatures.txt is at the root of the staged dataset. Name the dataset CPU GPU Input. See Data Management for the complete dataset upload procedure.

The workflow definition below uses ddi://~, so the input remains unselected until each workflow execution. To bind a workflow to a specific dataset instead, see Finding a Dataset DDI URI.

Create the Workflow#

Open Workflows from the main menu, select Custom Workflow and choose the YAML code editor. See Create Custom LEXIS Platform workflow for the complete portal procedure. Start with the following definition, which uses Karolina’s qcpu and qgpu node types as an example. For another HPC system, replace the location and node types with values supported by that system:

id: My_2_step_workflow
desc: Convert temperatures on a CPU node and average them on a GPU node
project_shortname: YOUR_PROJECT_SHORTNAME
jobs:

  Job1_Prepare:
    requirements:
      policy: preferred
      command_template_name: GenericCommandTemplate
      node_type_name: qcpu
      locations:
        - location_name: Karolina
          location_resource: YOUR_COMPUTATION_RESOURCE
      walltime_limit: 300
      max_cores: 128
      template_parameters:
        userScriptPath: YOUR_PROJECT_DIR/cpu-gpu-workflow/cpu_prepare.sh
      environment_variables: {}
    data_inputs:
      - source: ddi://~
        target: ./
    data_outputs:
      # Preserve the intermediate result as an iRODS dataset.
      - source: intermediate/
        target: ddi://~
        metadata:
          title: Fahrenheit temperatures
          access: project

  Job2_Compute:
    requirements:
      policy: preferred
      command_template_name: GenericCommandTemplate
      node_type_name: qgpu
      locations:
        - location_name: Karolina
          location_resource: YOUR_COMPUTATION_RESOURCE
      walltime_limit: 300
      max_cores: 16
      template_parameters:
        userScriptPath: YOUR_PROJECT_DIR/cpu-gpu-workflow/gpu_reduce.sh
      environment_variables: {}
    depends_on:
      - Job1_Prepare
    data_inputs: []
    data_outputs:
      - source: result/
        target: ddi://~
        metadata:
          title: Average temperature
          access: project
metadata:
  start_date: "2026-07-01T00:00:00.000Z"
  catchup: false

Review the translated workflow and click Create Workflow.

Run the Workflow#

Open the workflow details and click Create Workflow Execution. Check the selected HPC resource and input dataset, then create the execution. See Execution of LEXIS Platform Workflows for the complete execution procedure. LEXIS Platform first submits Job1_Prepare to a CPU node. After it finishes, it submits Job2_Compute to a GPU node.

Open each task in the execution graph to inspect its HPC job log. The CPU log reports how many temperatures were converted. The GPU log prints:

Average temperature: 63.680 F

Check the Results#

The execution has two output datasets in iRODS:

Fahrenheit temperatures contains the converted values in temperatures.txt.
Average temperature contains the calculated average in average.txt.

Two-Step CPU-GPU Workflow

Contents

Two-Step CPU-GPU Workflow #

Required Values#

Prepare the Cluster Scripts#

Prepare the Input Dataset#

Create the Workflow#

Run the Workflow#

Check the Results#