.. _tutorial_dynamical: ======================= If, while and branching ======================= Here we will introduce how if statements and while loops are represented in TaskBlaster by making use of branching. Since this is an advanced topic it is suggested that you first read about the conceptual difference between static and dynamical workflows in :ref:`explanation_dynamical`. A single if-statement --------------------- Consider following piece of pseudo code: :: a = A() if B(a): c = C(a) else: d = D(a) If A, B, C, and D are tasks, the outcome of B determines whether C or D will be executed. C and D will never both run, and you can't know which one will run until A and B are completed. So, this code needs to be part of a dynamic workflow. .. tbinit:: dynamical1 Let us now try to create the workflow above in TaskBlaster step by step. First initialize a repository, and write dummy tasks to tasks.py to represent tasks A, B, C, and D. For convenience, we also include task E for further tutorial in this page. .. tbshellcommand:: tb init .. tbfile:: tasks.py .. literalinclude:: tasks.py .. tbfile:: workflow.py And write the workflow corresponding to the pseudocode above. Note that at this stage of the tutorial, the exact Taskblaster syntax is irrelevant (it will be revisited below in more detail). Crucial thing here is that this is the exact Taskblaster representation of the pseudo code (for a graphical representation see :ref:`explanation_dynamical`). .. literalinclude:: workflow.py We have introduced two new decorators to tasks. Firstly, the ``@tb.branch()`` decorator, which will assign a task to a particular branch. If the decorator is omitted, the default branch will be ``entry``. Secondly, we utilized the ``@tb._if(true=, false=)`` decorator. The task with this decorator will now return a result, and whether that evaluates to True or False, a particular branch is chosen. Note the underscore in ``_if`` because ``if`` cannot be used since it is a reserved word. Same applies to lower case ``true`` and ``false`` parameters. We can now execute the workflow: .. tbshellcommand:: tb workflow workflow.py We see that only ``A`` and ``B`` tasks were created. We see this ``entry:`` text, which is the default branch for each task, if they are not given any other branch name by the ``@tb.branch`` decorator. We also see that B has some extra markings on it, ``if`` on the left hand side, and information about the control flow on the right hand side. The blue color of the True and False results indicate that this branch has not been decided yet (because the ``B`` "if-task" is not yet done). We can observe the current status of the project: .. tbshellcommand:: tb ls The above relates us back to the initial paragraph of the static workflow. The task dependency graph has now been generated as far as it can, before it encountered a branching statement. This is as far as static workflows can get us, thus *to execute one branch, is to execute a static workflow*. We can now run and observe the workflow: .. tbshellcommand:: tb run . .. tbshellcommand:: tb ls We note that the task to determine the branching is now ready, so we may continue with the workflow execution. .. tbshellcommand:: tb workflow workflow.py The workflow command will discover that the task associated with the if statement has now been evaluated, and it will make a jump. Therefore, it will execute the new branch called Dbranch, which is highlighted by making it green, and the rejected branch was made red. We also see, that due to creation of the Dbranch (in similar static sense as the entry-branch above) we obtain one new task, namely ``D``. We can now run this ``D`` task, to complete the workflow. .. tbshellcommand:: tb run . Running the workflow command has no further effect. .. tbshellcommand:: tb workflow workflow.py as all of the tasks are done. .. tbshellcommand:: tb ls Converging from multiple branches --------------------------------- Very often, we would like to take a detour with branching, and then return to the master control flow. In the following we will use the Collatz sequence as an example. The Collatz sequence is generated, starting from any positive number and following two simple rules: 1. If the number is even, divide by two 2. If the number is odd, multiply by three and add 1. The Collatz conjecture states that the sequence will always lead to the number 1. We will now recreate one iteration of the Collatz sequence (in the next section, we will iterate this further). Let's define very simple tasks. .. literalinclude:: tasks_collatz.py The pseudo code to do one iteration can be written as: :: def CollatzIteration(number, sequence): if is_even(number): new_number = divide_by_two(number) else: new_number = three_n_plus_one(number) return results(new_number, sequence) Let us write this workflow with Taskblaster. Following the same basic steps, we initialize the repository. .. tbinit:: converging_workflow .. tbshellcommand:: tb init .. tbfile:: tasks_collatz.py tasks.py .. tbfile:: converging_workflow.py Write the workflow for one iteration of the Collatz to the file `converging_workflow.py`: .. literalinclude:: converging_workflow.py We have introduced a new ``@tb.jump()`` keyword, which causes control flow to jump immediately from current branch to another. The usecases of jump include returning to main control flow after diverging into multiple branches. .. note:: One is allowed to have exactly one branching statement inside a branch, that is only one if or jump is allowed per branch. Below, is an illustration of `valid` branch and an `invalid` branch with two branching statement. :: @tb.branch('valid') @tb.jump('target') def taskA(self): ... @tb.branch('valid') @tb.jump('target') def taskB(self): ... @tb.branch('invalid') @tb.jump('target') def taskC(self): ... @tb.branch('invalid') @tb.if(true='target', false='otherbranch') def taskD(self): ... We have also introduced a new keyword, ``self.Phi(=, ...)``, which is used to select the correct branch for children Tasks with branching parent Tasks. ``Phi`` provides a mechanism to reference data before it exists, and resolve these inputs once the task has been completed, passing the correct branch to the children tasks. That is, when entering a branch with a task with a Phi-operator, the rule is that it will select the input routing based on which branch it came from. In this case, that is triggered by the jump statements. For the explicit example above the `gather_result` task will thus execute the function `results` with the input argument *number* given by the output from the `èven_task` if the *even_branch* was executed or by the output from the `odd_task` if the *odd_branch* was executed. Now run the workflow .. tbshellcommand:: tb workflow converging_workflow.py We can now run this task, and again, rerun the workflow to make a new iteration. We repeat these steps, until everything is done. .. tbshellcommand:: tb run . .. tbshellcommand:: tb workflow converging_workflow.py .. tbshellcommand:: tb run . .. tbshellcommand:: tb ls --sort=topo Let's see the output, which is 16 as expected, because input 5 is odd, we calculate 5*3+1=16. .. tbshellcommand:: tb view tree/gather_result We can illustrate the control flow and task parameter routing now with the following figure (Similar to :ref:`explanation_dynamical` the parameter routing is represented by the dashed green arrows and the control flow by the thick solid arrows). .. tbfile:: converging_style.json .. tbhiddenshellcommand:: tb view-workflow -s converging_style.json -f converging_workflow.py -o converging_workflow.html CollatzIteration .. tbviewworkflow:: converging_workflow.html .. raw:: html
Note how we in all cases jump to the last branch, which then collects the results. In the next section, we will go into even more complicated dynamical workflows, as we will finish the Collatz sequence. That is we will further use this dynamical workflow, as a dynamic subworkflow, on a dynamic workflow. Dynamical subworkflows inside dynamical workflows ------------------------------------------------- This will be the grande finale of the tutorial. We will utilize the workflow created above, the branching workflow, inside a workflow which demonstrates a while loop. .. tbinit:: grandefinale First initialize a new repository .. tbhiddenshellcommand:: tb init .. tbfile:: tasks_collatz.py tasks.py .. tbfile:: collatz2_workflow.py Then write the following workflow to the file *collatz2_workflow.py*: .. literalinclude:: collatz2_workflow.py As you can see the main workflow `CollatzSequence` now includes the `CollatzIteration` as a subworkflow. Note the keyword `loop` to the branch decorator in `CollatzSequence`. This keyword signifies that the branch can be revisited multiple times which is the way you make while loops in TaskBlaster. In the workflow `CollatzSequence` above first a single CollatzIteration will be executed. The task `sequence` adds the *number* to the list *sequence*. The first time the branch is visited the initial list sequence will just be the input argument, the next time the branch is visited *number* will be added to the previously computed sequence. This is controlled by `sequence=self.Phi(default=[self.number], entry=self.sequence)`. Finally the task `stop_iteration` checks if *number=1*, if so we are finished and the result (*sequence*) is collected by the task `result`, otherwise the *entry* branch is revisited for a new iteration. Let's see what happens when we execute this workflow: .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo .. tbshellcommand:: tb workflow collatz2_workflow.py && tb run . > /dev/null && tb ls -csirITfo --sort=topo Note how each iteration of the while loop is run seperately, since it involves resolving an if statement. The *loop* keyword to the *branch* decorator automatically numbers the folders so that each iteration of the branch is given a specific folder. Finally the result points to the sequence calculated in the final iteration. .. tbshellcommand:: tb view tree/result .. tbfile:: converging_style.json .. tbhiddenshellcommand:: tb view-workflow -s converging_style.json -f collatz2_workflow.py -o grande_finale_workflow.html CollatzSequence .. tbviewworkflow:: grande_finale_workflow.html The graphical representation of the workflow is given :download:`here <../../_static/grande_finale_workflow.html>`. Note, that due to experimental nature of the automatic workflow visualizer, some variable arrows are missing, but the branching arrows are all there.