.. _create_lexis_workflow_from_definition: ================================== LEXIS Workflows :updated:`flag` ================================== The LEXIS Workflow is a template that specifies everything that is needed to actually run the computation. The LEXIS Worflow is created from your uploaded job script or container and it might also require some data that you need to have prepared beforehand. You can visit :ref:`user-dataset-creation` section if you require help with performing the uploads. As for the LEXIS Workflows, you can find dedicated section in main Menu of the **LEXIS Portal**. .. figure:: img/howto/workflows_wayto.png :width: 768 :align: center :alt: Give me some workflow You will immediately see all the workflows assigned to selected project. The *Type* column can quickly tell you what kind of workflow it is. ----------------------------------------------------- Create LEXIS workflow from job script ----------------------------------------------------- Once you have your job script prepared and uploaded, it is time to transform it to executable template. This is done from the *Workflows* section where the *HPC Jobscript* button is waiting. .. figure:: img/howto/create_workflow_custom_job_script_way_to_.png :width: 768 :align: center :alt: Find me job script Click it and you will get a list of all job scripts attached to your project. Locate the one you wish to turn into a workflow and click on the little *Down* arrow. You will get the list of all versions of the particular job script. Hit the *Create Workflow* button of the version you want. .. figure:: img/howto/create_workflow_custom_job_script_button.png :width: 768 :align: center :alt: Job script paradise A new form opens where you are required to input basic information, starting with the name you want to give to this spectacular workflow and some description. .. figure:: img/howto/create_workflow_jobscript_1.png :width: 768 :align: center :alt: One step toward madness In the second step, you must choose one of the assigned resources to the project along with execution cluster and partition. .. figure:: img/howto/create_workflow_jobscript_2.png :width: 768 :align: center :alt: Two steps in chaos During the third step, there is an option to *Enable input dataset staging* and *Enable output dataset staging*. You should check the first checkbox if your job script requires input dataset. You must have this dataset already uploaded into your project. The dataset will be staged to */input* directory relative to the HPC job execution context. It is possible you might have multiple input dataset for this workflow. You can add as many as needed with the *Add Another Dataset Input* button. Similarly, if your job script produces a dataset, check the second checkbox. Fill in the required information. You have options to produce multiple datasets as well. .. figure:: img/howto/create_workflow_jobscript_3.png :width: 768 :align: center :alt: Three steps through hell Fourth page is for advanced settings. *Walltime limit* is the maximum time the job will stay in submitted state before it is killed by the system. *Max cores* is used to limit the number of cores used. .. figure:: img/howto/create_workflow_jobscript_4.png :width: 768 :align: center :alt: Four steps towards your doom And finally, you will get the short summary just to be sure you entered important things correctly. .. figure:: img/howto/create_workflow_jobscript_4_summary.png :width: 768 :align: center :alt: Did I win Confirm with *Create* button and you will be brought back to the *Workflows* section. Your new workflow will appear in the list, but it may take some seconds. This is a great time to get some coffee. .. figure:: img/howto/created_jobscript_worfklow_.png :width: 768 :align: center :alt: Created Worfklow in list yay ----------------------------------------------------- Create LEXIS workflow from container ----------------------------------------------------- Once you have your container prepared and uploaded, it is time to transform it to executable template. This is done from the *Workflows* section where the *Container* button is waiting. .. figure:: img/howto/create_workflow_custom_container_way_to_.png :width: 768 :align: center :alt: Find me container Click it and you will get collection of blocks representing all containers attached to your project. Locate the one you wish to turn into a workflow and click on the tempting *Create Workflow* button. .. figure:: img/howto/create_workflow_container_button.png :width: 768 :align: center :alt: Container paradise A new form opens where you required to input basic information, starting with the name you want to give to this spectacular workflow and some description. .. figure:: img/howto/create_workflow_container_1.png :width: 768 :align: center :alt: One step toward joy In the second step, you must choose one of the assigned resources to the project along with execution cluster and partition. .. figure:: img/howto/create_workflow_container_2.png :width: 768 :align: center :alt: Two steps in happiness During the third step, there is an option to *Enable input dataset staging* and *Enable output dataset staging*. You should check the first checkbox if your container requires input dataset. You must have this dataset already uploaded into your project. The dataset will be staged to */input* directory relative to the HPC job execution context. It is possible you might have multiple input dataset for this workflow. You can add as many as needed with the *Add Another Dataset Input* button. Similarly, if your container produces a dataset, check the second checkbox. Fill in the required information. You have options to produce multiple datasets as well. .. figure:: img/howto/create_workflow_container_3.png :width: 768 :align: center :alt: Three steps through paradise In the fourth step, you can specify the parameters of your container, if there are any. .. figure:: img/howto/create_workflow_container_4.png :width: 768 :align: center :alt: Four steps above rainbow Fifth page is for advanced settings. *Walltime limit* is the maximum time the job will stay in submitted state before it is killed by the system. *Max cores* is used to limit the number of cores used. .. figure:: img/howto/create_workflow_container_5.png :width: 768 :align: center :alt: Five steps towards your fortune And finally, you will get the short summary just to be sure you entered important things correctly. .. figure:: img/howto/create_workflow_container_6_summary.png :width: 768 :align: center :alt: Did I lose Confirm with *Create* button and you will be brought back to the *Workflows* section. Your new workflow will appear in the list, but it may take some seconds. Maybe go for a tea? .. figure:: img/howto/created_container_worfklow_.png :width: 768 :align: center :alt: Created Worfklow in list yay ----------------------------------------------------- Create Custom LEXIS workflow ----------------------------------------------------- You have option to create completely custom workflow. To start, locate the *Custom Workflow* button in the *Workflows* section under the main Menu. .. figure:: img/howto/create_workflow_custom_workflow_way_to_.png :width: 768 :align: center :alt: Find me customs Custom LEXIS workflows are specified declaratively using LEXIS Workflow Definition. For complete documentation, see `Lexis Workflow Definition <../lexis_orchestration_languages/lexis_workflow_definition.html>`_. In total, you have three options how to approach the definition. 1. Define workflow from YAML and upload it directly. 2. Define workflow from YAML using the code editor. 3. Use the external *Workflow Builder*. .. figure:: img/howto/i_have_options.png :width: 768 :align: center :alt: Find me customs As for the first option, you need to provide your own workflow in a YAML file. The third option will open a new tab that will lead you to interactive builder. The second option allows you to use the API directly. Example of high-level workflow declaration using LEXIS Worfklow Definition: .. code-block:: yaml :linenos: id: test_Easy_WF2_improved desc: Test Workflow in LWD project_shortname: testproject jobs: hpcJob_node1: requirements: command_template_name: TestTemplate node_type_name: qcpu_exp locations: - location_name: Karolina location_resource: DD-XX-X walltime_limit: 3600 max_cores: 128 policy: preferred template_parameters: inputParam: LoremIpsum - Value data_inputs: - source: ddi://926658e6-xxxx-xxxx-xxxx-xxxxxxx target: inputDir1 data_outputs: - source: inputDir1/ target: ddi://~ metadata: title: LWD Test intermediate output $name: input_to_output_dataset access: project additionalMetadata: LEXIS_dataset_type: TESTING_DATASET - source: inputDir1/ metadata: $name: input_to_output_dataset hpcJob_node2: requirements: command_template_name: TestTemplate node_type_name: qcpu_exp locations: - location_name: Karolina location_resource: DD-XX-X walltime_limit: 3600 max_cores: 128 policy: preferred template_parameters: inputParam: LoremIpsum - Value data_inputs: - source: ddi://926658e6-xxxx-xxxx-xxxx-xxxxxxx target: inputDir1 - source: job://hpcJob_node1/input_to_output_dataset/ target: inputDir2 data_outputs: [] metadata: start_date: "2025-10-20T11:10:12.880Z" catchup: false .. figure:: img/howto/creating_lexis_wf.png :width: 768 :align: center :alt: Customary workflow Whatever method you choose, continue with the *Create Workflow* button and the workflow will be created. It might take some time before appearing in the list of your workflows, so how about hot cocoa? -------------------------------------- Creating workflow from HPC Application -------------------------------------- For most users, the presented three options of creating workflows should be enough. However, your project might require specific approach and if it happens to have its own definition, it will be resting in this section. .. _executions-of-LEXIS-Workflows: ----------------------------- Execution of LEXIS Workflows ----------------------------- Specific run of your **LEXIS Workflow** is called execution. No matter what method you used to create your workflow, you can now execute it to compute it on our clusters. You can simply start the execution of your workflow from the *Workflows* section. Find the desired workflow and click on the *Detail* button next to it. .. figure:: img/howto/executions_WF_.png :width: 768 :align: center :alt: Execute order 66 This will display a lot of useful information about your workflow. In the upper part, there are tabs *Details*, *Advanced* and *Workflow Graph Definition*. The *Advanced* tab contains a lot of detailed information about your workflow. In the *Workflow Graph Definition* you can find the graph structure of the workflow. In the *Details*, you get basic information like the name of your workflow, to what project it belongs and its description. There is also the list containing previous executions of your workflow made either by you or by the members of your project. There is also the state of that execution and you can reveal more details with the *Details* button. The most important part is the pair of buttons, *Create Workflow Execution* and *Delete*. The latter is there to get rid of the workflow should its usefulness has run its course. The former starts the execution of your workflow. Click on it. .. figure:: img/howto/executions_WF_createnew_excecution_.png :width: 768 :align: center :alt: Start the firework A new form will appear. First thing to do is to enter a name for this specific execution. Then you must configure the job. If your workflow requires data inputs and outputs, you specify them here. .. figure:: img/howto/wf_create_new_excecution_.png :width: 768 :align: center :alt: Preparation phase The *Requirements* tab allows you to configure the parameters concerning the run itself. You already entered this information during the creation of the workflow, but you can change them here if you need to. Once you are satisfied, you can press the *Create* button. .. figure:: img/howto/wf_create_new_excecution2_.png :width: 768 :align: center :alt: Preparation phase numero duo You will be returned back to the *Details* section of your workflow and your new execution will be added to the list as waiting in the queue. .. figure:: img/howto/running_workflow_landing_.png :width: 768 :align: center :alt: Back to beginning After a while, the status will change to running. .. figure:: img/howto/running_workflow_execution.png :width: 768 :align: center :alt: Run to beginning And it will eventually end. Anytime after you create the execution, you can check its details. .. figure:: img/howto/running_workflow_execution_detail.png :width: 768 :align: center :alt: Run to line There will be a *Details* tab that, among other things, contains the *Re-Execute* button to quickly run the same job. There is another tab labeled *Progress* where you can monitor as your workflow slowly lights up and is completed in the form of interactive graph. .. figure:: img/howto/running_workflow_graph_.png :width: 768 :align: center :alt: running_workflow_graph You can try moving your cursor over one of the boxes to get more information tied to this particular step. And if you want to know more, the *Show/Hide* button is for you. The *View HPC Job Logs* button contains even more knowledge. .. figure:: img/howto/running_workflow_detail_.png :width: 768 :align: center :alt: running_workflow_detail You might not like the vertical view and just for you, there is a horizontal view as well. For some larger workflows, the graph will feature groups, so it is as crystal clear as possible. Additionally, your awesome Workflow execution graph can be downloaded and saved to your computer, so you can admire it anywhere! .. figure:: img/howto/running_workflow_options_.png :width: 768 :align: center :alt: running_workflow_options The workflow uses various colours to indicate the states of the **Workflow Taks**. You can find the legend below. .. figure:: img/howto/states_of_wf_legend.png :width: 768 :align: center :alt: states_of_wf_legend .. _command_templates_manage: ----------------------------------------- Command Templates' management ----------------------------------------- Whenever you create your new project, there are four default command templates you can use. Additionally, as a project owner, you can limit which command templates are available for other members in your project. To do that, go to *Projects*, press the *Details* button and hit the *Manage Applications* button. .. figure:: img/howto/manage_templates_wayto.png :align: center :alt: way to manage templates button The switch buttons you can spot at first glance are doing exactly what you expect. You can use them to turn on and off the specific command templates for the members of your project. But there is the *Computing Resource* dropdown that is actually very important. .. figure:: img/howto/manage_templates_settings.png :align: center :alt: way to manage templates button Every command template is specific for **given pairing of Computing resource and the Project**. You can have two projects with same computing resource and both projects can have different command templates enabled. And conversely, you can have two or more computing resources assigned to a single project and for each of those resources you can enable different combination of the command templates.