.. _how_to_full_project:

How to set up a large scale project with Taskblaster?
=====================================================

All examples in web pages utilize the most basic building blocks of taskblaster,
such as the local ``tasks.py`` file. Here we set up a complete Python project,
which is installable, and has your workflows. This also enables advanced properties 
of Taskblaster such as custom object encoding and decoding.

Download and execute (in a new folder) the following :download:`script <setup_full_project.sh>`, 
which will create necessary files for a skeleton of a Python Taskblaster project 
and a python virtual environment, where it pip installs it as editable.
You may also want to run this script step by step to ensure that each step works.

.. tbinit:: full_project

.. tbfile:: setup_full_project.sh 

.. tbshellcommand:: chmod +x setup_full_project.sh && ./setup_full_project.sh

The script will create a following file and directory structure

.. tbshellcommand:: cd tb_demo_venv/tb_demo && find . | grep -v "egg-info"

There is a Python module called ``tb_demo`` in the folder with the same name.
It is identified as a module, because it has the ``__init__.py`` file,
Taskblaster can find it, because it is installed, since we did call 
``pip install -e .`` in its parent folder with the appropriate 
``setup.py`` (which signifies an installable package).

Take a look of the ``pyproject.toml`` of the project. 
Note our package depends on Taskblaster, and thus running the pip install command will automatically install Taskblaster
to our virtual environment from ``pip`` repository.
If you want to use your own gitlab cloned taskblaster, comment out the `'taskblaster'` dependency,
and create the virtual environment with `--system-site-packages` to include your own taskblaster installation
(or alternativaly `pip install -e .` taskblaster in your virtual environment as well.

.. tbshellcommand: cat tb_demo_venv/tb_demo/pyproject.toml

We may now initialize Taskblaster with the projects module. Instead of ``tb init`` we use
the syntax ``tb init [MODULE]``.

You can activate the virtual environment now:

.. tbvenv:: tb_demo_venv/bin/activate

.. tbshellcommand:: tb init tb_demo

.. warning::

    Before writing ``tb init tb_demo`` make sure to go to a fresh folder (do not create a Taskblaster
    repository to your home folder, or to your virtual environment.

You may examine the tasks and the workflow. The new feature enabled by associating Taskblaster
with its designated project module is enabling of user encoding. Here we introduce the easier way,
which is to create Python classes with `tb_decode` and `tb_encode` methods. This is how one
can pass through any instances of classes as parameters to tasks, and as outputs of tasks.

.. tbshellcommand:: cat tb_demo_venv/tb_demo/tb_demo/mysubmodule/__init__.py

There are wide usecases for such custom classes. For example, one may use classes to contain
parameters for calculations, but only make them actually perform certain steps of calculations.
The tasks used in this demo reflect this ability in the simplest possible manner.

.. tbshellcommand:: cat tb_demo_venv/tb_demo/tb_demo/mysubmodule/tasks.py

The custom encoding and decoding also allows much easier changing of parameters, 
as the user controls the encoring and decoding.

We can run the workflow.

.. tbshellcommand:: tb workflow tb_demo_venv/tb_demo/tb_demo/main_workflow.py

And execute it's tasks.

.. tbshellcommand:: tb run .

It is perhaps insightful to look at the actual ``output.json`` of the ``create_object`` task.

.. tbshellcommand:: cat tree/create_object/output.json

And we see how the module path to the class has been stored, and that is how Taskblaster will know how to search for the right encoder/decoder.

.. tbvenv:: deactivate