.. _howto_guide: ============= How-to Guide ============= We would like to briefly introduce you to the LEXIS platform by following paragraphs. The LEXIS platform offers various actions for the users interested to usage of HPC and cloud via graphical interface or HTTP API. Within the other possibilities, following actions are available on the platform: - create LEXIS projects - request access to existing LEXIS projects - manage members of LEXIS projects as project manager - create and manage users datasets - manage LEXIS project computation resources - create LEXIS workflows with containerized applications - create LEXIS workflows with HPC command applications - manage LEXIS custom workflows - execute LEXIS workflows - see LEXIS workflow execution progress - browse LEXIS workflow execution results .. _platform-registration: -------------------- Registration -------------------- Registration is so easy. Upon clicking the login button located in the upper right corner, you will encounter two sign-up options. The `B2Access `_ and `MyAccessID - Puhuri `_ are trustful authentication authorities supporting eduGAIN, ORCID and Social Identities. .. image:: FirstStepsImages/fsteps_sign_in_options.png :target: ../../_images/fsteps_sign_in_options.png :width: 320 If you don’t have any projects in LEXIS yet, you can create a new one or request access to an existing project. Both of these options will be described in the following chapters. .. _lexis-project-creation: -------------------------- Creation of LEXIS Project -------------------------- :ref:`LEXIS Project` is the basic organization unit. If you don’t have access to any project yet, you will see the option to create a new one immediately after logging into LEXIS. .. figure:: img/access_or_create_project.png :width: 240 :align: center :alt: access or create project If you already have access to projects but need to create a new one, this can be done in the *Projects* in the navigation menu using the button *Create Project*. .. figure:: img/create_project_wayto.png :width: 768 :align: center :alt: Create project button Next, it is necessary to add in the details about the new project. .. figure:: img/create_project_form.png :width: 360 :align: center :alt: Create project form The newly created project can be viewed in the project list in the *Projects* menu. .. figure:: img/list_of_projects.png :width: 768 :align: center :alt: List of projects .. _requesting-access-existing-project: -------------------------------------------- Requesting access to existing LEXIS Project -------------------------------------------- If you don’t have access to any project yet, you can request access to an existing project immediately after logging in. .. figure:: img/access_or_create_project.png :width: 240 :align: center :alt: Access or create project If you already have access to Lexis Projects but need to request access to another existing project, this can be done in the *Projects* tab using the button *Request Project Access*. .. figure:: img/request_project_access_wayto.png :width: 768 :align: center :alt: Request project access buttom To submit a request, it is sufficient to know the short name of the project. .. figure:: img/request_project_access.png :width: 240 :align: center :alt: Request project access The request will be sent to the Project Owner, who must approve access to the project and assign you the appropriate role in it (:ref:`manage-members-as-project-manage`). Next you will receive an email in response with information on whether you have been granted access to the LEXIS Project or not. .. _requesting-to-LEXIS-resources: ------------------------------------------------- Requesting to LEXIS project computation resources ------------------------------------------------- Execution of workflows and storing files requires a :ref:`LEXIS Computation Resource`. To assign one with computation time and one with data storage allocation, click on Resources next to the created project on the *Projects*. .. figure:: img/Resources_button_in_project.png :width: 360 :align: center :alt: Resources_button_in_project You can request the allocation of resources in the project list using the *Resources* button. .. figure:: img/project_resources_button.png :width: 768 :align: center :alt: Project_resources_button This will display a list of resources already allocated to the LEXIS Project, and there you will find the *Request Resources Access* button. The right to request Resources within the LEXIS Project is only granted to users in the role of Project Manager or Project Owner. .. figure:: img/request_resources_access_button_listresources.png :width: 768 :align: center :alt: request_resources_access_button_listresources Click on *Request Resource Access* and fill out the form. Please provide the Resources name and the email of the PI for the selected project. .. figure:: img/resource_form2.png :width: 240 :align: center :alt: resource_form After your request has been successfully submitted, you will be informed about the progress via your email. Now wait until the PI of the resource accepts your request. If you are the PI, an email will be sent to you with a one-time link to accept the request. To assign a storage resource, fill in the resource name as “iRODS LEXIS V2”, choose the provider “IT4Innovations”, and fill in the email field with “support@lexis-project.eu”. After the request has been processed by the PI, the applicant will be notified by email whether access to the requested resources has been accepted or rejected. .. _user-dataset-creation: ------------------------- Creation of Users Dataset ------------------------- Before executing some computation on LEXIS Platform, you would like probably to upload some input data - :ref:`dict-users-dataset`. To do that, navigate to *Data Sets* and *Uploads* section via navigation menu on the left side. You will find yourself on page with all accessible uploaded data sets list. .. figure:: img/upload_dataset_button_wayto.png :width: 768 :align: center :alt: upload_dataset_button Clicking on green button *Upload Dataset*, following form will appear. .. figure:: img/upload_dataset_form.png :width: 768 :align: center :alt: upload_dataset_form For uploading data an existing assigned :ref:`LEXIS Computation Resource` to a :ref:`LEXIS Project`. Access defines visibility scope of dataset to one of user (dataset owner), project (all members in the project), public. Every dataset can have metadata. .. figure:: img/upload_dataset_metadata.png :width: 768 :align: center :alt: upload_dataset_metadata .. figure:: img/upload_dataset_summarization.png :width: 768 :align: center :alt: upload_dataset_summarization .. _modify-users-dataset: -------------------------------- Modify of Users Data Set -------------------------------- The list of already uploaded Users Data Sets can be viewed in *Data Sets/Uploads*. The allowed actions for a specific dataset can be displayed using the *Actions* button. .. figure:: img/actions_datasets.png :width: 768 :align: center :alt: actions_datasets Commonly used actions for the User Data Set include: *Copy ID*, *Copy PATH*, *File List*, *Delete* and *Downland*. .. figure:: img/detail_of_dataset.png :width: 768 :align: center :alt: detail_of_dataset You can use or get inspired by publicly available Users Data Sets. Here they are available for preview and possible download. .. figure:: img/public_datasets.png :width: 768 :align: center :alt: public_datasets .. _using-container-applications: ---------------------- Container Applications ---------------------- One of the supported methods for delivering instructions for a Workflow in the LEXIS Project is recording them in an :ref:`dict-hpc-container-application`. You can add a Container in the *Applications*/*Containers* menu. Then, in the tabs, select the LEXIS project for which you want to add the container and click on *Create Container* button. .. figure:: img/create_container_wayto.png :width: 768 :align: center :alt: create_container_wayto Now, you need to upload the file with the container's contents, an icon, enter the name, and gradually fill in additional information about the new container. .. figure:: img/create_new_conteiner_form_add.png :width: 768 :align: center :alt: create_new_conteiner_form_add .. figure:: img/create_new_conteiner_form_params.png :width: 360 :align: center :alt: create_new_conteiner_form_params At the end, a summary will be displayed for review and to complete the saving of the new container. .. figure:: img/create_new_conteiner_form_result.png :width: 768 :align: center :alt: create_new_conteiner_form_result Existing containers are displayed in blocks, including basic information. By clicking on each block, you can create a new Workflow with selected container. .. figure:: img/container_show_all.png :width: 768 :align: center :alt: container_show_all .. _using-hpc-command-applications: ------------------------- HPC Command applications ------------------------- Another option for defining an input template for a Workflow is preparation (compiled) by specialist an :ref:`dict-hpc-command-application`. If you have your own HPC application ready, please contact support@lexis-project.eu, who will ensure the addition of your HPC application to the LEXIS Project. After the new HPC application is uploaded, it will appear in the list of already uploaded applications in the *Applications*/*HPC Application*. module. By clicking on a specific application, you can continue configuring the Workflow and then run it with the specified parameters. .. figure:: img/HPC_Applications_view.png :width: 768 :align: center :alt: HPC_Applications_view .. figure:: img/create_wf_hpc_app.png :width: 360 :align: center :alt: create_wf_hpc_app .. figure:: img/create_wf_hpc_app_2.png :width: 360 :align: center :alt: create_wf_hpc_app_2 .. figure:: img/create_wf_hpc_app_3.png :width: 360 :align: center :alt: create_wf_hpc_app_3.png .. figure:: img/create_wf_hpc_app_4_5.png :width: 360 :align: center :alt: create_wf_hpc_app_4_5 .. _about-custom-hpc-jobs: ----------------------- Custom HPC Jobs ----------------------- To create user HPC Jobscript, navigate to *Applications* and *Custom HPC Jobs*. Click on green button *Create Jobscript*. .. figure:: img/create_jobscript_button_wayto.png :width: 768 :align: center :alt: create_jobscript_button_wayto Use following code as :ref:`example job script ` and continue. .. _example-custom-job-script: .. code-block:: bash # Example Job Script for LEXIS Workflow source /cvmfs/software.eessi.io/versions/2023.06/init/bash ls ./input # in this directory should appear staged input dataset cat ./input/vinice-geojson.json # reads content of uploaded file to dataset echo "I am running!!" .. figure:: img/custom_job_script.png :width: 768 :align: center :alt: custom_job_script .. figure:: img/custom_job_script_wf_summary.png :width: 768 :align: center :alt: custom_job_script_wf_summary Now the job script for workflow is prepared and workflow can be created by clicking *Create Workflow* next to the prepared job script in *Custom HPC Jobs* page. .. figure:: img/create_workflow_custom_job_script.png :width: 768 :align: center :alt: create_workflow_custom_job_script .. figure:: img/create_workflow_jobscript_1.png :width: 768 :align: center :alt: create_workflow_jobscript_1 In second step, one of assigned resources to the project is selected together with execution cluster and partition. .. figure:: img/create_workflow_jobscript_2.png :width: 768 :align: center :alt: create_workflow_jobscript_2 The data input is enabled and default dataset is selected. The dataset will be staged to */input* directory relative to HPC job execution contex. .. figure:: img/create_workflow_jobscript_3.png :width: 768 :align: center :alt: create_workflow_jobscript_3 .. figure:: img/create_workflow_jobscript_4_summary.png :width: 768 :align: center :alt: create_workflow_jobscript_4_summary After creation of workflow, it will finally appear after a few seconds in *Custom HPC Script Worfklows* section in *Custom HPC Jobs* page. .. figure:: img/created_jobscript_worfklow.png :width: 768 :align: center :alt: Created Worfklow in list Now, it is ready to create first execution. Click on green *Execute* button and fill the form. .. figure:: img/execution_form.png :width: 768 :align: center :alt: Workflow execution form In the execution detail, there is visualisation. Particular steps produce logs, which are accessible by moving the cursor over the step and clicking on the *Logs* button. .. figure:: img/execution_logs_failed.png :width: 768 :align: center :alt: How to find execution logs on failure Logs from HPC job can usually be found in *...heappe_job_waiting* task's log. .. figure:: img/workflow_job_log.png :width: 768 :align: center :alt: HPC job logs .. _final-job-log: # Job log - partial GeoJSON content and "I am running" message .. code-block:: pypylog [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - [ [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - 16.64590904709604, [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - 48.810181792975015 [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - ], [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - [ [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - 16.64727894500959, [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - 48.81075254030276 [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - ] [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - ], [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - "type": "LineString" [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - } [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - } [2024-07-18T11:58:00.642+0000] {logging_mixin.py:188} INFO - ], [2024-07-18T11:58:00.643+0000] {logging_mixin.py:188} INFO - "bbox": null [2024-07-18T11:58:00.643+0000] {logging_mixin.py:188} INFO - } [2024-07-18T11:58:00.643+0000] {logging_mixin.py:188} INFO - I am running!! [2024-07-18T11:58:00.650+0000] {base.py:293} INFO - Success criteria met. Exiting. [2024-07-18T11:58:00.672+0000] {taskinstance.py:1138} INFO - Marking task as SUCCESS. .. _about-custom-lexis-worflows: ---------------------- Custom LEXIS Workflows ---------------------- Another option for specifying a template with tasks and dependencies is defining a User Workflow using a YAML script. To add a new User Workflow, select *Application*/*User Workflow* on the left. Then, the *Create Workflow* button will appear on the right. .. figure:: img/create_user_WF_wayto.png :width: 768 :align: center :alt: create_user_WF_wayto The YAML script for the new Workflow can either be uploaded from your local machine or created using the form in the provided editor. .. figure:: img/upload_yaml_user_WF.png :width: 768 :align: center :alt: upload_yaml_user_WF .. figure:: img/editor_yaml_user_WF.png :width: 768 :align: center :alt: editor_yaml_user_WF Now you need to enter the name of the new workflow and choose a short name for the LEXIS Project. After successfully creating the Workflow, you can immediately execut and test the new Workflow. In the execution detail, there is visualisation. Particular steps produce logs, which are accessible by moving the cursor over the step and clicking on the *Logs* button. .. _executions-of-LEXIS-Workflows: ----------------------------- Executions of LEXIS Workflows ----------------------------- LEXIS Workflows can always be executed immediately after entering the input application and dataset within the *Applications* module. Subsequently, all executed LEXIS Workflows can be viewed in the *Workflows* module. Here, there is also the option to execute the workflow again with modified input parameters. So, if you need to view the details of an already executed workflow, you can do it using the magnifying *Glass* button in the row of the specific workflow or the *Executions* button. .. figure:: img/executions_WF.png :width: 768 :align: center :alt: executions_WF In the *Details* tab using the blue button *Workflow Executions*, you can view the list of all executions of this Workflow, including the details of each run. With the green button *Create Workflow Execution*, you can set the parameters for a new execution and start it. .. figure:: img/executions_WF_createnew_excecution.png :width: 768 :align: center :alt: executions_WF_createnew_excecution .. figure:: img/wf_create_new_excecution.png :width: 360 :align: center :alt: wf_create_new_excecution In the Workflow *Graph* tab, you can view the graph of individual tasks in the selected LEXIS Workflow. .. figure:: img/WF_graph.png :width: 768 :align: center :alt: WF_graph Below, we provide a legend for the colors used in the Workflow to indicate different statuses of Workflows Tasks - :ref:`LEXIS Task`. .. figure:: img/states_of_wf_legend.png :width: 768 :align: center :alt: states_of_wf_legend .. _users-dashboard: --------------- Users Dashboard --------------- The portal features a Dashboard module that provides a convenient overview of running workflows, their progress, and results. In the *Dashboard*/*Workflow*, this information can be viewed in clear diagrams, including some details about the workflows. .. figure:: img/dashboard_workflow.png :width: 768 :align: center :alt: dashboard_workflow .. _project-permissions-management: ---------------------------- Users Permissions Management ---------------------------- In the top right corner, you can view the details of your user account. .. figure:: img/view_profile.png :width: 240 :align: center :alt: view_profile By clicking on *View Profile*, a table of roles for each of your projects will be displayed on the right. .. figure:: img/role_in_projects.png :width: 768 :align: center :alt: role_in_projects Within the LEXIS Project, we define 4 user roles. Below is a list and description of each: **Project Owner** - This role is automatically assigned to the user who creates a new LEXIS Project. There can be only one Owner per project. The user in this role has the highest level of permissions on the project and is responsible for approving new users and assigning them appropriate roles. **Project Manager** - The Owner can assign this role to other users on the project to delegate project and user management to additional users. Multiple users can hold the Manager role within a project. **Project Member** - This role is for standard users on the project. A user in this role can work with Workflows and DataSets but cannot publish any DataSets. **Project DataPublisher** - A user in this role can work with Workflows and DataSets, and also has the permission to publish Data Sets. .. _manage-members-as-project-manage: --------------------------------------------------- Manage members of LEXIS projects as project manager --------------------------------------------------- If someone :ref:`requesting-access-existing-project` where you are in the role of Project Owner or Project Manager, a new request will appear here for your approval. If you approve the request, the new user will automatically be assigned the role of Project Member. .. figure:: img/project_acces_requests_keynew_click.png :width: 360 :align: center :alt: project_acces_requests_keynew_click If you want to set a higher permission level for the user (role Project Manager) or deny access to the project, this can be done in the *Project* module, under the *Users* button. .. figure:: img/project_users_button.png :width: 768 :align: center :alt: project_users_button .. figure:: img/make_manager_unnassign_user.png :width: 768 :align: center :alt: make_manager_unnassign_user