How to connect multiple workflows?#
Here is an example of how one can generate multiple workflows, and then generate one more workflow
which would combine them. The key point is to realize that the workflow instances are created in the def workflow function
give access to task references, which can then further on supplied to the workflows which combine these workflows.
Below is an example, which creates two sets of workflows, and then creates one more set of workflows which are taken to be the product of these workflows.
Create a following script workflow.py:
import taskblaster as tb
@tb.workflow
class Molecule:
atoms = tb.var()
@tb.task
def relax(self):
return tb.node('define', obj=self.atoms)
@tb.workflow
class NanoParticle:
atoms = tb.var()
@tb.task
def relax(self):
return tb.node('define', obj=self.atoms)
@tb.workflow
class Combined:
molecule = tb.var()
nanoparticle = tb.var()
@tb.task
def relax(self):
return tb.node('define', obj=[self.molecule, self.nanoparticle])
def workflow(runner):
mworkflows = []
molecules = ['H2O', 'H2']
for molecule in molecules:
_runner = runner.with_subdirectory(molecule)
wf = Molecule(atoms=molecule)
mworkflows.append((molecule, wf))
_runner.run_workflow(wf)
nworkflows = []
for nanoparticle in ['Ag', 'Fe', 'Mg']:
_runner = runner.with_subdirectory(nanoparticle)
wf = NanoParticle(atoms=nanoparticle)
nworkflows.append((nanoparticle, wf))
_runner.run_workflow(wf)
from itertools import product
for (mname, mwf), (nname, nwf) in product(mworkflows, nworkflows):
_runner = runner.with_subdirectory(mname + nname)
_runner.run_workflow(
Combined(molecule=mwf.relax, nanoparticle=nwf.relax)
)
We can now run the workflow and observe that we have crosstabulated the “molecules” and the “nanoparticles”.
$ tb init
Created repository using module "taskblaster.repository" in "/home/me/repo".
$ tb workflow workflow.py entry: add new 0/0 tree/H2O/relax entry: add new 0/0 tree/H2/relax entry: add new 0/0 tree/Ag/relax entry: add new 0/0 tree/Fe/relax entry: add new 0/0 tree/Mg/relax entry: add new 0/2 tree/H2OAg/relax entry: add new 0/2 tree/H2OFe/relax entry: add new 0/2 tree/H2OMg/relax entry: add new 0/2 tree/H2Ag/relax entry: add new 0/2 tree/H2Fe/relax entry: add new 0/2 tree/H2Mg/relax add: 11
$ tb run .
Starting worker rank=000 size=001
[rank=000 2026-04-17 12:58:35 N/A-0/1] Worker class: —
[rank=000 2026-04-17 12:58:35 N/A-0/1] Required tags: —
[rank=000 2026-04-17 12:58:35 N/A-0/1] Supported tags: —
[rank=000 2026-04-17 12:58:35 N/A-0/1] name: None
tags: —
required_tags: —
resources: None
max_tasks: None
subworker_size: None
subworker_count: None
wall_time: None
no_mpi: None
workflow: None
[rank=000 2026-04-17 12:58:35 N/A-0/1] Main loop
[rank=000 2026-04-17 12:58:35 N/A-0/1] Running Ag/relax ...
[rank=000 2026-04-17 12:58:35 N/A-0/1] Task Ag/relax finished in 0:00:00.000387
[rank=000 2026-04-17 12:58:35 N/A-0/1] Running Fe/relax ...
[rank=000 2026-04-17 12:58:35 N/A-0/1] Task Fe/relax finished in 0:00:00.000302
[rank=000 2026-04-17 12:58:35 N/A-0/1] Running H2/relax ...
[rank=000 2026-04-17 12:58:35 N/A-0/1] Task H2/relax finished in 0:00:00.000244
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running H2Ag/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task H2Ag/relax finished in 0:00:00.000274
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running H2Fe/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task H2Fe/relax finished in 0:00:00.000239
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running Mg/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task Mg/relax finished in 0:00:00.000222
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running H2Mg/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task H2Mg/relax finished in 0:00:00.000341
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running H2O/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task H2O/relax finished in 0:00:00.000249
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running H2OAg/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task H2OAg/relax finished in 0:00:00.000244
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running H2OFe/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task H2OFe/relax finished in 0:00:00.000248
[rank=000 2026-04-17 12:58:36 N/A-0/1] Running H2OMg/relax ...
[rank=000 2026-04-17 12:58:36 N/A-0/1] Task H2OMg/relax finished in 0:00:00.000229
[rank=000 2026-04-17 12:58:36 N/A-0/1] No available tasks, end worker main loop
$ tb ls -cosirITf output state info tags worker time folder ──────────────────────── ──────── ────────── ─────────── ─────────── ─────────── ────────────────────────────────────────────────── 'Ag' done 0/0 N/A 00:00:00 tree/Ag/relax 'Fe' done 0/0 N/A 00:00:00 tree/Fe/relax 'H2' done 0/0 N/A 00:00:00 tree/H2/relax ['H2', 'Ag'] done 2/2 N/A 00:00:00 tree/H2Ag/relax ['H2', 'Fe'] done 2/2 N/A 00:00:00 tree/H2Fe/relax ['H2', 'Mg'] done 2/2 N/A 00:00:00 tree/H2Mg/relax 'H2O' done 0/0 N/A 00:00:00 tree/H2O/relax ['H2O', 'Ag'] done 2/2 N/A 00:00:00 tree/H2OAg/relax ['H2O', 'Fe'] done 2/2 N/A 00:00:00 tree/H2OFe/relax ['H2O', 'Mg'] done 2/2 N/A 00:00:00 tree/H2OMg/relax 'Mg' done 0/0 N/A 00:00:00 tree/Mg/relax