Workflow generator and for loops
================================
Now we will investigate how to generate subworkflows in a loop.
There are in principle two cases one can encounter:

1. The parameters that one wants to loop over are known and given by user input.

2. The parameters are determined from the result of another task.

Case 1 only requires a static workflow, while case 2 is dynamic in the sense that the tasks in the loop cannot be generated until the initial task that determines the parameters to loop over is done. Since an implementation that can handle the second case also can handle the first case all loops in TaskBlaster are dynamic. For-loops are implemented using the so called `dynamic workflow generator`. Below we show a simple example:

Start by inititializing a TaskBlaster repo:

.. tbinit:: wf_generator

.. tbshellcommand:: tb init

Make two simple tasks in `tasks.py`:

.. tbfile:: tasks.py

.. literalinclude:: tasks.py
   :start-after: first_part_start
   :end-before: first_part_end

Now we want to create a workflow that takes a number as input, and uses the task `plus_two` to determine a list of numbers to loop over. The task `ok` should then be executed for all numbers in the list.

First we add the task `plus_two` to the file `workflow.py`:

.. tbfile:: workflow.py

.. literalinclude:: workflow.py

and run the workflow

.. tbshellcommand:: tb workflow workflow.py
.. tbshellcommand:: tb run tree

Verify using `tb view` that you have the expected output from the task.

.. tbshellcommand:: tb view .

Now you are ready to add the loop. Add the following to the `workflow.py` file and save it to a new file `workflow2.py`.

.. tbfile:: workflow2.py

.. literalinclude:: workflow2.py

We have now added a `dynamical_workflow_generator` to the workflow. This decorator takes an input argument which specifies how the result is collected.
The input argument is a dictionary where the keys corresponds to the name that will be given to the results task and the item is the path to the results
that will be collected by the results task. It is possible to specify any number of seperate results collections. Here we simply collect all results to the results task `results`. The workflow generator returns a `dynamical_workflow_generator_task` that needs to be specified in the `tasks.py` file or alternatively imported from an external package.

In this tutorial we will for simplicitly just  update the `tasks.py` file, which is automatically imported by TaskBlaster, even though it is a bit unintuitive to have workflows in this file:

.. literalinclude:: tasks.py

You can notice that we added a `dynamical_workflow_generator_task` with the name `generate_wfs_from_list`. Since each generated subworkflow will be given a specific folder the workflow generator task

1. takes a list as input
2. loops over the list
3. Defines which workflow that should be generated and what input arguments it should have and saves it to the variable wf.
4. Defines a unique name for the subfolder for the generated workflow
5. Yields the name of the subfolder together with the wf.

This is a minimal example for how you can use a workflow generator, but in principle you can more advanced logic inside the function (e. g. yielding different subworkflows depending on the input arguments etc.). Finally we also need to define the workflow `SubWorkflow` that should be generated. Here we have a simple workflow with two simple tasks, but it is possible to generate arbitrary complex workflows. 

Now let's see what happens when we run the workflow.

.. tbshellcommand:: tb workflow workflow2.py
		    
.. tbshellcommand:: tb ls

As you can see two tasks were created, the initializer `init` and the finalizer which we defined as `results`.
Let's try to run the initializer:

.. tbshellcommand:: tb run tree/generated_wfs/init

.. tbshellcommand:: tb ls

You can now see that three workflows were generated, each containing two tasks. The dependencies of the results task were also correctly updated to 0/6. We can now run the remaining tasks:

.. tbshellcommand:: tb run tree

.. tbshellcommand:: tb ls

As you can see all tasks are now in the state `done` and you can view the results

.. tbshellcommand:: tb view tree/generated_wfs/results


Running everything in a single go
----------------------------------
In the previous example we did the calculations step by step. It is instructive to see what happens if we try to run everything
in a single go. First remove the tree using `tb remove tree` or alternatively make a new repository in a new folder and copy the files
`tasks.py` and `workflow2.py` to the new folder:

.. tbshellcommand:: tb remove tree --force

Double check so that all tasks were removed

.. tbshellcommand:: tb ls

Now run the workflow and then run all tasks

.. tbshellcommand:: tb workflow workflow2.py

.. tbshellcommand:: tb run tree

.. tbshellcommand:: tb ls

As you can see all tasks were generated and run directly.

Using multiple finalizers
-------------------------
We will now see how to collect results from different tasks in different results tasks.
First remove the tree:

.. tbshellcommand:: tb remove tree --force

Modify the input to the `dynamical_workflow_generator` decorator in `workflow2.py` and save to a new file `workflow3.py`

.. tbfile:: workflow3.py

.. literalinclude:: workflow3.py
   :start-after: start_display
   :end-before: end_display

We have now defined three seperate finalizers, one pointing to all generated tasks, and two pointing to generated_task1 and generated_task_2 respectively.
Run the workflow and run all tasks.

.. tbshellcommand:: tb workflow workflow3.py

.. tbshellcommand:: tb run tree

.. tbshellcommand:: tb ls

You can view the new results tasks using `tb view`

.. tbshellcommand:: tb view tree/generated_wfs/results_task1

As you can see the finalizer results_task1 only points to the output of the tasks `generated_task1` etc.