.. _cpu-gpu-two-step-workflow:

#####################################
Two-Step CPU-GPU Workflow :new:`flag`
#####################################

In this tutorial, you create a reusable two-step workflow. A CPU job converts
temperatures from Celsius to Fahrenheit, then a GPU job calculates their
average and saves the results to iRODS. Set up the cluster scripts once, then
run the workflow repeatedly from the **LEXIS Platform** with a different input
dataset for each execution.

Both jobs use ``GenericCommandTemplate`` to run scripts stored on the shared
project filesystem.

.. note::
   You need a **LEXIS Platform** project with CPU and GPU resources, SSH access to the
   selected HPC system and permission to write to its shared project directory.

===============
Required Values
===============

Before starting, identify the values that you will substitute throughout this
tutorial:

``YOUR_PROJECT_SHORTNAME`` and ``YOUR_COMPUTATION_RESOURCE``
   Use the project and computation resource identifiers described in
   :ref:`custom-workflow-values`. If separate resources provide the CPU and GPU
   node types, use the appropriate resource name in each YAML job.

``YOUR_PROJECT_DIR``
   The full path to a shared project directory on the selected HPC system.
   On Karolina, for example, use ``/scratch/project/PROJECT_ID``. Store the
   workflow scripts and shared intermediate files in this directory. Do not
   use your home directory: workflow jobs are launched by a service account
   that can access **LEXIS Platform** project directories, but not your personal home
   directory.

===========================
Prepare the Cluster Scripts
===========================

On your local computer, create a folder named ``cpu-gpu-workflow`` for the
four script files:

.. code-block:: bash

   mkdir cpu-gpu-workflow
   cd cpu-gpu-workflow

The module commands below use the module names available on Karolina. When
using another HPC system, replace them with the commands that provide Python
and CuPy on that system.

First create a wrapper that loads Python before starting the CPU program:

.. code-block:: bash

   #!/usr/bin/env bash
   set -e

   # Load Python
   module load Python

   # Run the CPU program
   python3 YOUR_PROJECT_DIR/cpu-gpu-workflow/cpu_prepare.py

   # Copy its output to the shared project directory for the GPU job
   mkdir -p YOUR_PROJECT_DIR/cpu-gpu-workflow/intermediate
   cp intermediate/temperatures.txt YOUR_PROJECT_DIR/cpu-gpu-workflow/intermediate/

Save it as ``cpu_prepare.sh`` in the local ``cpu-gpu-workflow`` folder.

The CPU script converts every temperature from Celsius to Fahrenheit and
writes the result to the job's ``intermediate`` directory.
The wrapper then copies the same result to the shared project directory for
the GPU job:

.. code-block:: python

   #!/usr/bin/env python3

   from pathlib import Path

   celsius = [
       float(line)
       for line in Path("temperatures.txt").read_text().splitlines()
   ]
   fahrenheit = [temperature * 1.8 + 32 for temperature in celsius]
   output = "\n".join(str(temperature) for temperature in fahrenheit) + "\n"

   output_dir = Path("intermediate")
   output_dir.mkdir(exist_ok=True)
   (output_dir / "temperatures.txt").write_text(output)

   print(f"Converted {len(celsius)} temperatures")

Save it as ``cpu_prepare.py`` in the same local folder.

The GPU wrapper loads CuPy, copies the intermediate file into the job's
working directory and starts the GPU program:

.. code-block:: bash

   #!/usr/bin/env bash
   set -e

   # Load CuPy
   module load CuPy

   # Copy the CPU output into this job's working directory
   cp YOUR_PROJECT_DIR/cpu-gpu-workflow/intermediate/temperatures.txt .

   # Run the GPU program
   python3 YOUR_PROJECT_DIR/cpu-gpu-workflow/gpu_reduce.py

Save it as ``gpu_reduce.sh``. The wrapper runs this CuPy program:

.. code-block:: python

   #!/usr/bin/env python3

   from pathlib import Path

   import cupy as cp


   temperatures = cp.loadtxt("temperatures.txt")
   average = float(cp.mean(temperatures).get())

   output_dir = Path("result")
   output_dir.mkdir(exist_ok=True)
   result = f"Average temperature: {average:.3f} F\n"
   (output_dir / "average.txt").write_text(result)

   print(result, end="")

Save it as ``gpu_reduce.py`` in the same local folder. The folder should now
contain:

.. code-block:: text

   cpu-gpu-workflow/
   |-- cpu_prepare.py
   |-- cpu_prepare.sh
   |-- gpu_reduce.py
   `-- gpu_reduce.sh

Return to the directory containing ``cpu-gpu-workflow`` and upload the whole
folder. Replace ``USERNAME`` with your cluster username and
``YOUR_PROJECT_DIR`` with the full shared project directory. The following
example uses Karolina; use the login address of your selected HPC system when
running elsewhere:

.. code-block:: bash

   cd ..
   scp -r cpu-gpu-workflow USERNAME@karolina.it4i.cz:YOUR_PROJECT_DIR/

Finally, log in to the system and make the uploaded scripts executable:

.. code-block:: bash

   ssh USERNAME@karolina.it4i.cz
   cd YOUR_PROJECT_DIR/cpu-gpu-workflow/
   chmod 750 *.sh *.py

.. _cpu-gpu-input-dataset:

=========================
Prepare the Input Dataset
=========================

Create a folder named ``cpu-gpu-input`` containing ``temperatures.txt``.
Enter one temperature in degrees Celsius per line:

.. code-block:: text

   12
   15.5
   18
   20
   22.5

Create the archive from inside the folder so that it does not add an extra
directory level:

.. code-block:: bash

   cd cpu-gpu-input
   zip ../cpu-gpu-input.zip temperatures.txt

Upload ``cpu-gpu-input.zip`` in *Data Management/Datasets* and enable
unpacking, so that ``temperatures.txt`` is at the root of the staged dataset.
Name the dataset ``CPU GPU Input``. See
:ref:`user-dataset-creation` for the complete dataset upload procedure.

The workflow definition below uses ``ddi://~``, so the input remains
unselected until each workflow execution. To bind a workflow to a specific
dataset instead, see :ref:`finding-dataset-ddi-uri`.

===========================
Create the Workflow
===========================

Open *Workflows* from the main menu, select *Custom Workflow* and choose the
YAML code editor. See :ref:`create-custom-lexis-workflow` for the complete
portal procedure. Start with the following definition, which uses Karolina's
``qcpu`` and ``qgpu`` node types as an example. For another HPC system, replace
the location and node types with values supported by that system:

.. code-block:: yaml

   id: My_2_step_workflow
   desc: Convert temperatures on a CPU node and average them on a GPU node
   project_shortname: YOUR_PROJECT_SHORTNAME
   jobs:

     Job1_Prepare:
       requirements:
         policy: preferred
         command_template_name: GenericCommandTemplate
         node_type_name: qcpu
         locations:
           - location_name: Karolina
             location_resource: YOUR_COMPUTATION_RESOURCE
         walltime_limit: 300
         max_cores: 128
         template_parameters:
           userScriptPath: YOUR_PROJECT_DIR/cpu-gpu-workflow/cpu_prepare.sh
         environment_variables: {}
       data_inputs:
         - source: ddi://~
           target: ./
       data_outputs:
         # Preserve the intermediate result as an iRODS dataset.
         - source: intermediate/
           target: ddi://~
           metadata:
             title: Fahrenheit temperatures
             access: project

     Job2_Compute:
       requirements:
         policy: preferred
         command_template_name: GenericCommandTemplate
         node_type_name: qgpu
         locations:
           - location_name: Karolina
             location_resource: YOUR_COMPUTATION_RESOURCE
         walltime_limit: 300
         max_cores: 16
         template_parameters:
           userScriptPath: YOUR_PROJECT_DIR/cpu-gpu-workflow/gpu_reduce.sh
         environment_variables: {}
       depends_on:
         - Job1_Prepare
       data_inputs: []
       data_outputs:
         - source: result/
           target: ddi://~
           metadata:
             title: Average temperature
             access: project
   metadata:
     start_date: "2026-07-01T00:00:00.000Z"
     catchup: false

Review the translated workflow and click *Create Workflow*.

================
Run the Workflow
================

Open the workflow details and click *Create Workflow Execution*. Check the
selected HPC resource and input dataset, then create the execution. See
:ref:`executions-of-LEXIS-Workflows` for the complete execution procedure.
**LEXIS Platform** first submits ``Job1_Prepare`` to a CPU node. After it finishes, it
submits ``Job2_Compute`` to a GPU node.

Open each task in the execution graph to inspect its HPC job log. The CPU log
reports how many temperatures were converted. The GPU log prints:

.. code-block:: text

   Average temperature: 63.680 F

=================
Check the Results
=================

The execution has two output datasets in iRODS:

* ``Fahrenheit temperatures`` contains the converted values in
  ``temperatures.txt``.
* ``Average temperature`` contains the calculated average in ``average.txt``.