Command-line reference

Taskblaster provides a command-line interface based on a single toplevel command, normally installed as tb, with a number of subcommands. Each command has a --help page. The help page of the toplevel command is:

$ tb --help
Usage: tb [OPTIONS] COMMAND [ARGS]...

  Taskblaster, a high-throughput workflow utility.

  Utility to define and run vast quantities of computational tasks, organizing
  inputs and outputs in a directory tree.

  Use the init subcommand to initialize an empty repository.  Then write a
  workflow function and run it to create a directory tree of tasks.  Then
  submit tasks using your favourite high-performance computing batch system.

Options:
  --help  Show this message and exit.

Commands:
  completion     Print bash command-line completion incantation.
  graph          Generate dependency graph.
  info           Print information about repository.
  init           Initialize repository inside directory.
  ls             List tasks under directory TREEs.
  merge          Merge repositories (possibly calculated on a different...
  registry       View or manipulate registry.
  remove         Delete tasks in TREE entirely.
  resolve        Mark conflicts as resolved in TREE.
  run            Launch worker to execute tasks.
  special        More rare special commands.
  stat           Print statistics about selected tasks.
  submit         Mark tasks in TREE and dependencies for execution.
  tag            List, tag, or untag tasks in TREE.
  unresolve      Mark resolved tasks as in conflict.
  unrun          Delete output files from TREE and reset task state to new.
  view           View detailed information or execute task-specific actions.
  view-workflow  Write HTML files from a specified workflow class.
  warden         Use Warden to handle failures.
  workers        View or manipulate workers.
  workflow       Run workflow creating folders for tasks inside tree.

The page exhaustively lists the documentation for to all subcommands and their options across all --help pages.

tb

Taskblaster, a high-throughput workflow utility.

Utility to define and run vast quantities of computational tasks, organizing inputs and outputs in a directory tree.

Use the init subcommand to initialize an empty repository. Then write a workflow function and run it to create a directory tree of tasks. Then submit tasks using your favourite high-performance computing batch system.

Usage

tb [OPTIONS] COMMAND [ARGS]...

completion

Print bash command-line completion incantation.

To enable command-line completion, include the script the shell rc file, e.g., ~/.bashrc.

For shells other than bash, see the documentation of click for how to set up command-line completion.

Usage

tb completion [OPTIONS]

graph

Generate dependency graph.

This computes the dependency graph of the specified tasks and prints it in machine-friendly graphviz format for further processing. Examples:

$ tb graph | dot -T svg > graph.svg # convert to svg using graphviz

$ tb graph | dot -T pdf > graph.pdf

$ tb graph | display # open window using imagemagick display command

Usage

tb graph [OPTIONS] [TREE]...

Arguments

TREE

Optional argument(s)

info

Print information about repository.

This prints a brief overview of special files and directories associated with the current project.

Usage

tb info [OPTIONS]

init

Initialize repository inside directory.

The optional MODULE argument can be used to specify a plugin. A plugin is a Python module defining a subclass of the taskblaster Repository class. This can be used to provide a custom JSON encoder to store objects not known to taskblaster, and to enable parallel workers and subworkers using MPI.

Usage

tb init [OPTIONS] [MODULE] [DIRECTORY]

Arguments

[MODULE]

Optional argument

DIRECTORY

Optional argument

ls

List tasks under directory TREEs.

Find tasks inside specified TREEs and collect their dependencies whether inside TREE or not. Then perform the specified actions on those tasks and their dependencies.

Usage

tb ls [OPTIONS] [TREE]...

Options

--parents

List ancestors of selected tasks outside selection. Implies topological sort (default: False).

-c, --columns <columns>

Columns to display: M: monitor, p: priority, d: depth, f: folder, s: state, i: info, I: myqueue id and subworker, r: tags, t: time info, T: time duration, c: conflict, C: conflict info, o: output, Z: frozen info. Default: sirITpf. Use + to mean in addition to default columns, e.g. -c+o. Frozen tasks are marked by ❄ and conflicts with C in the info string.

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

--ready

Select only tasks with all done dependencies.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--sort <sort>

Sort tasks alphabetically (‘name’) or topologically (‘topo’).

Options:

name | topo

--order <order>

Order rows according to column string given.

-F, --failure <failure>

Select only tasks with failure string matching the string

--monitor <monitor>

File or a glob pattern to monitor when displaying M-column. Applies only to running tasks. Will display the last line of the last modified file matching this pattern.

Arguments

TREE

Optional argument(s)

merge

Merge repositories (possibly calculated on a different HPC platforms)

To start the merge, do tb merge start [--new] DESTINATION_REPO. Then add files using tb merge add as you like. You can use tb merge ls to list the current status of the merge, and to also observe the differences. The tb merge select [--dest/--source] command is used to select from where the things should be added from. After you have added all the tasks you want to merge, one commits the merge to the other repository with tb merge commit. The repository goes now into copy-stage, and one has to write tb merge copy in order to finalize the merge. One can launch multiple instances of tb merge copy to expedite the copying of files. After all copies are done, you can stop the merge by doing tb merge stop.

Usage

tb merge [OPTIONS] COMMAND [ARGS]...

abort

Abort the merge, no changes will be saved.

Only applicable is no copy has been started. If copy has been started it is too late.

Restore the backup of the repo.

Usage

tb merge abort [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

add

Add new tasks to the merge

Usage

tb merge add [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

--silent

Do not print to screen.

--progress

Display a progress bar. Implies silent.

--tags <TAGS>

Only add tasks (and their ancestors) if it has one of these flags

Arguments

TREE

Optional argument(s)

cache

Cache all non-cached differences.

Usage

tb merge cache [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--progress

Display a progress bar. Implies silent.

commit

Start merge procedure.

Usage

tb merge commit [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--force

Unrun tasks without prompting for confirmation.

--silent

Do not print to screen.

--progress

Display a progress bar. Implies silent.

Arguments

TREE

Optional argument(s)

copy

Run a single worker to copy files.

Multiple copy commands can be launches simultaneously.

Usage

tb merge copy [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--reset

Reset all of the syncing and copy failed merge states back to committed. Do not launch a copy worker.

--progress

Display a progress bar. Implies silent.

--lfs-rule <lfs_rule>

Selection criteria for the file to be applicable for the large filesupport. Rules will be separated by colon. For example, “2M:.gpw” would select files larger than 2MiB or ones matching the pattern `.gpw`. Sizes are indicated by an integer and a letter from KMGT.

--lfs <lfs>

Large file support:

no-lfs: Everything will be copied.

move-to-dest: Large files will be moved from source to dest, and thus will vanish from the source. Source will be invalid after this, so use this option only if you want to destroy source.

move-to-dest-symlink-source: Large files will be moved from source to dest, and the source will have symlinks pointing to the files in dest. Source will be fragile after this, so use this option only if you can trust that destination will not be changed.

symlink-dest: No large files will be moved, but dest will contain symlinks to the large source files. Destination will be fragile after this, so use this option only if you can trust that source will not be changed.

Options:

move-to-dest-symlink-source | no-lfs | move-to-dest | symlink-dest

info

Print inportant info about the merge process going on.

Usage

tb merge info [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

ls

Display listing on the staged tasks

Usage

tb merge ls [OPTIONS] [TREE]...

Options

--parents

List ancestors of selected tasks outside selection. Implies topological sort (default: False).

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

--sort <sort>

Sort tasks alphabetically (‘name’) or topologically (‘topo’).

Options:

name | topo

--diff <diff>

Select only tasks which differ by md: missing in destination, ie: I equals, id: I differ, ieom: I equals O missing in dst, idom: I differ O missing in dst, idoe: I differ O equals, ieod: I equals O differ, ioe: IO equals, iod: I differ O differ where I is input and O is output.

Options:

md | ie | id | ieom | idom | idoe | ieod | ioe | iod

-d, --dstate <dstate>

Select only tasks with this state. State be any of: ‘nqrdFpC’ or M for missing. Select based on state on the destination repository.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

-F, --failure <failure>

Select only tasks with failure string matching the string

-S, --select <select>

Which merge select state is taken, select -SS (source) -SD (destination)

--merge-status <merge_status>

Status of the merge: eg. --merge-status smF. a (added), c (committed), s (copying/syncing), m (merged), F (failed)

Arguments

TREE

Optional argument(s)

select

Select tasks to be either from the source or destination repository.

This command is used to change the selected origin for each task on the merge. Use TREE or qualifiers to select larger sets of tasks.

By default, tb merge select will select source on all of the tasks given by the qualifiers and TREE. However, users may explictly also give --source parameter to highlight this.

To select a destination for a given task, use tb merge select --dest TREE.

Usage

tb merge select [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--dest

Select destination for the given tasks.

--source

Select source for the given tasks.

--diff <diff>

Select only tasks which differ by md: missing in destination, ie: I equals, id: I differ, ieom: I equals O missing in dst, idom: I differ O missing in dst, idoe: I differ O equals, ieod: I equals O differ, ioe: IO equals, iod: I differ O differ where I is input and O is output.

Options:

md | ie | id | ieom | idom | idoe | ieod | ioe | iod

-F, --failure <failure>

Select only tasks with failure string matching the string

-S, --select <select>

Which merge select state is taken, select -SS (source) -SD (destination)

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

-d, --dstate <dstate>

Select only tasks with this state. State be any of: ‘nqrdFpC’ or M for missing. Select based on state on the destination repository.

Arguments

TREE

Optional argument(s)

start

Start merge procedure.

Usage

tb merge start [OPTIONS] DESTINATIONREPOPATH

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--new

Create a new repository.

Arguments

DESTINATIONREPOPATH

Required argument

stat

Usage

tb merge stat [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

TREE

Optional argument(s)

stop

Stop the merge.

Usage

tb merge stop [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

unadd

Unadd added tasks of the merge

Usage

tb merge unadd [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

--tags <TAGS>

Only unadd tasks (and their descendants) if it has one of these flags

--diff <diff>

Select only tasks which differ by md: missing in destination, ie: I equals, id: I differ, ieom: I equals O missing in dst, idom: I differ O missing in dst, idoe: I differ O equals, ieod: I equals O differ, ioe: IO equals, iod: I differ O differ where I is input and O is output.

Options:

md | ie | id | ieom | idom | idoe | ieod | ioe | iod

-d, --dstate <dstate>

Select only tasks with this state. State be any of: ‘nqrdFpC’ or M for missing. Select based on state on the destination repository. Use M to specify that the task can be missing in the destination repo.

--silent

Do not print to screen.

--progress

Display a progress bar. Implies silent.

-F, --failure <failure>

Select only tasks with failure string matching the string

-S, --select <select>

Which merge select state is taken, select -SS (source) -SD (destination)

Arguments

TREE

Optional argument(s)

registry

View or manipulate registry.

Usage

tb registry [OPTIONS] COMMAND [ARGS]...

ancestors

Usage

tb registry ancestors [OPTIONS]

build

Add all tasks in the tree to the registry.

Usage

tb registry build [OPTIONS]

ls

Usage

tb registry ls [OPTIONS]

patch-serialized-inputs

Patch all tasks so inputs are stored in the registry.

The purpose of this command is to migrate repositories where input lives only inside input.json such that the input will be inside the registry. input.json will not be deleted. The operation prompts for confirmation.

This does not affect tasks that already have inputs stored in registry.

Usage

tb registry patch-serialized-inputs [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--unapply

Reverse operation: Remove inputs from registry (mostly for testing). Does not prompt for confirmation.

remove

Delete tasks in TREE entirely. Caution is advised.

Usage

tb remove [OPTIONS] [TREE]...

Options

-z, --dry-run

Simulate what would happen, but do nothing.

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

--force

Remove tasks without prompting for confirmation.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

TREE

Optional argument(s)

resolve

Mark conflicts as resolved in TREE.

Change the conflict state to “resolved” for selected tasks with conflict state “conflict”.

Usage

tb resolve [OPTIONS] [TREE]...

Options

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

Arguments

TREE

Optional argument(s)

run

Launch worker to execute tasks.

The worker runs tasks in TREE and any dependencies with matching tags. TREE defaults to all queued tasks.

Usage

tb run [OPTIONS] [TREE]...

Options

--subworker-count <subworker_count>

Number of MPI subworkers in run.

--subworker-size <subworker_size>

Number of processes in each MPI subworker.

--greedy

Run also tasks created while running specified selection.

--workflow <workflow>

After all work is done, run the workflow given as a parameter (can be a Python import path or a file). Effectively specifying the workflow parameter, worker can generate work for itself infinitely on a dynamical workflow.

--worker-class <WORKER>

Worker class for this worker. The name must exist in the resource configuration, see tb workers config.

--tags <TAGS>

Allow worker to pick up tasks with any of TAGS specified as comma-separated list.

--require <TAGS>

Require worker to pick up only tasks with all the TAGS specified as comma-separated list.

--max-tasks <NUM>

Maximum number of tasks for worker to run.

--ready

Select only tasks with all done dependencies.

-z, --dry-run

Simulate what would happen, but do nothing.

--wall-time <wall_time>

Maximum time for worker to run. Worker terminates when walltime is exceeded or would be exceeded within task-wall-time. Default is unlimited time.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

TREE

Optional argument(s)

special

More rare special commands.

Usage

tb special [OPTIONS] COMMAND [ARGS]...

amend

Retrospectively change the output of a done task.

Usage

tb special amend [OPTIONS] IDENTIFIER TASK [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

-z, --dry-run

Simulate what would happen, but do nothing.

--explanation <explanation>

A long mandatory explanation what this amendment does.

Arguments

IDENTIFIER

Required argument

TASK

Required argument

TREE

Optional argument(s)

freeze

Freeze tasks in TREE. The freeze recursively affects all task descendants.

Usage

tb special freeze [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

TREE

Optional argument(s)

heuristics

Perform heuristics to registry to find out possible inconsistencies.

Currently peforms following check:

• Ensures that topological_depth table meets all FOREIGN_KEY conditions to registry table. This is for older taskblaster versions, before the FOREIGN_KEY constraint was introduced.

Usage

tb special heuristics [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

lsw

Usage

tb special lsw [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

-c, --columns <columns>

Columns to display: n: name, i: importpath, I: serialized_input, S: source. Default: ni. Use + to mean in addition to default columns, e.g. -c+o.

make-directories

Create all directories which exists currently in the registry

Usage

tb special make-directories [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

rename-import-path

Rename import path for tasks.

Replace OLD import paths with NEW ones. OLD and NEW are function import paths as specified to tb.node(). The primary use of this command is to keep the repository up to date when a function has been moved or renamed during refactoring.

No attempt is made to ensure that the new and old functions are equivalent or to otherwise invalidate tasks, so caution is advised.

Usage

tb special rename-import-path [OPTIONS] OLD NEW [TREE]...

Options

--force

Rename targets without prompting for confirmation.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

OLD

Required argument

NEW

Required argument

TREE

Optional argument(s)

rename-object

Rename a serializable object.

Taskblaster objects are stored by storing the import path of a serializable object. If this object’s import path is moved, this special function may be used to change all of the paths to these objects to new ones.

Usage

tb special rename-object [OPTIONS] OLD NEW [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--force

Rename objects without prompting for confirmation.

Arguments

OLD

Required argument

NEW

Required argument

TREE

Optional argument(s)

repair

Repairs the registry from (relatively) safe errors.

There errors include things, such as foreign key mismatches of auxilliary tables (used only for speed and listings).

To see full list of repairs, see:

tb special heuristics --help

Usage

tb special repair [OPTIONS]

Options

--force

Repair registry without prompting for confirmation.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

run-workflow-clobber-implicit

Run workflow and overwrite implicit dependencies willy-nilly.

This command resolves a conflict by overwriting the original input with the new (conflicting) input for any selected tasks with conflicts.

Please be careful: Abusing this command can make results non-reproducible. It should be used to resolve conflicts arising from internal format changes, or fixes where an input must be changed although the resulting computation is the same.

This command only affects selected tasks that have unresolved conflicts.

Usage

tb special run-workflow-clobber-implicit [OPTIONS] [SCRIPT]

Options

-m, --module <module>

-z, --dry-run

Simulate what would happen, but do nothing.

--silent

Do not print to screen.

Arguments

SCRIPT

Optional argument

task-name-hash

Usage

tb special task-name-hash [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

unfreeze

Unfreeze tasks in TREE. This affects only tasks that were frozen due to the corresponding tb freeze command. To unfreeze tasks that are frozen due to a conflict, resolve the conflict.

Usage

tb special unfreeze [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

TREE

Optional argument(s)

validate-input-json

Usage

tb special validate-input-json [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--repair

Repair the input.json from the database

Arguments

TREE

Optional argument(s)

wait

Wait until all tasks are done or that there are no workers anymore.

Useful for tutorials and automatic bash scripting of Taskblaster.

Tree argument can be given, and then stats are only counted for that.

Usage

tb special wait [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--interval <interval>

How many seconds to wait before the next check.

Default:

60

Arguments

TREE

Optional argument(s)

workflow

Run whole workflow or parts of it.

Without TREE, run master workflow.

With TREE, rerun the source workflow of every task in TREE.

Usage

tb special workflow [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

TREE

Optional argument(s)

stat

Print statistics about selected tasks.

Usage

tb stat [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

Arguments

TREE

Optional argument(s)

submit

Mark tasks in TREE and dependencies for execution.

Only affects new tasks. To submit a failed task, unrun it first.

Usage

tb submit [OPTIONS] [TREE]...

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--priority <priority>

Assign priority to the submitted tasks.

Arguments

TREE

Optional argument(s)

tag

List, tag, or untag tasks in TREE.

Without options, list all tasks or selected tasks with their tags.

With –add or –untag apply actions on selected tasks only.

Usage

tb tag [OPTIONS] [TREE]...

Options

--add <TAG>

Add TAG to tasks.

--untag <TAG>

Remove TAG from tasks.

-F, --failure <failure>

Select only tasks with failure string matching the string

Arguments

TREE

Optional argument(s)

unresolve

Mark resolved tasks as in conflict.

Change the conflict state to “conflict” for selected tasks with conlict state “resolved”.

Usage

tb unresolve [OPTIONS] [TREE]...

Options

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

Arguments

TREE

Optional argument(s)

unrun

Delete output files from TREE and reset task state to new.

Unrunning a task also unruns its descendants.

Usage

tb unrun [OPTIONS] [TREE]...

Options

-s, --state <state>

Select only tasks with this state. State be any of: ‘nqrdFpC’.

-F, --failure <failure>

Select only tasks with failure string matching the string

--force

Unrun tasks without prompting for confirmation.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--only-descendants

Only unrun the descendants. This option is only useful in case the user wants to amend done tasks, and then no done descendants are allowed in this case.

--resubmit

Resubmit all tasks which are unrun.

--rerun

Rerun all tasks which are unrun. Make sure the tasks are small enough to be run on a single core on this machine.

--no-remove

Do not remove tasks due whose implicit dependencies are unrun. Normally, tasks that depend on if-statements, generators, etc., are removed when their parent constructrs are unrun. This flag disables that behaviour.

--conflict-states <conflict_states>

Only unrun tasks which have one of the conflict states (n=none, r=resolved, c=conflict). For example, –conflict-states=rc

--filter-task <filter_task>

Usage: –filter-task=FUNCTION. Runs a task returning a boolean whether this task should be selected. The function should look like def filter_task(record): return True/False. Note that this filter only applies to tasks directly in the glob pattern and not their descendants, as they need to be unrun anyway.

Arguments

TREE

Optional argument(s)

view

View detailed information or execute task-specific actions.

Usage

tb view [OPTIONS] [TREE]...

Options

--action <action>

Perform specified action for the selected tasks. To associate tasks with actions, see the documentationfor the @tb.actions decorator.

Arguments

TREE

Optional argument(s)

view-workflow

Write HTML files from a specified workflow class.

Example usage:

tb view-workflow package.workflows.TestWorkflow

In case workflow cannot be imported from a package, user can also specify the file that contains the workflow as:

tb view-workflow -f workflow.py TestWorkflow

Usage

tb view-workflow [OPTIONS] [WORKFLOW_CLASS]

Options

-o, --output <output>

-f, --file <file>

-s, --style <style>

-b, --browser

Arguments

WORKFLOW_CLASS

Optional argument

warden

Use Warden to handle failures.

Usage

tb warden [OPTIONS] COMMAND [ARGS]...

handler-rollback

XXX if you use a handler, this will be used to undo the last attempt the handler made.

Usage

tb warden handler-rollback [OPTIONS] [TREE]...

Arguments

TREE

Optional argument(s)

kick

Kick failed tasks to the warden’s handler added after failure occurs.

Usage

tb warden kick [OPTIONS] [TREE]...

Options

--tags <TAGS>

Select tasks with any of these tags

Arguments

TREE

Optional argument(s)

update-task-inputs

User override of the task’s inputs.

Usage

tb warden update-task-inputs [OPTIONS] [TREE]...

Options

--tags <TAGS>

Select tasks with any of these tags

Arguments

TREE

Optional argument(s)

workers

View or manipulate workers.

Usage

tb workers [OPTIONS] COMMAND [ARGS]...

config

View worker configuration.

Usage

tb workers config [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

ls

List taskblaster workers.

Usage

tb workers ls [OPTIONS]

Options

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

-c, --columns <columns>

Columns to display: i: myqueue_id, s: status, W: wall time, q: queue, t: submit time, S: start time, e: end time. Default: isWqt. Use + to mean in addition to default columns, e.g. -c+o.

submit

Submit workers as myqueue jobs.

Without arguments, submit one generic worker. If WORKER is a number, submit that many generic workers. Else, WORKER must be the name of a configured worker in the resources file from which settings are taken (see tb workers config). Submit NUM workers of that type, or one worker if NUM is not specified.

Examples:

tb workers submit # submit one generic worker

tb workers submit 4 # submit 4 generic workers

tb workers submit worker1 # submit one worker of type ‘worker1’

tb workers submit worker1:4 # submit 4 workers of type ‘worker1’

tb workers submit –workflow mymodule.workflows.main 1

Options below override options taken from resources file.

Usage

tb workers submit [OPTIONS] [WORKER[:NUM]...]

Options

-z, --dry-run

Simulate what would happen, but do nothing.

--subworker-count <COUNT>

Number of MPI subworkers in run.

--subworker-size <SIZE>

Number of processes in each MPI subworker.

-R, --resources <resources>

Resource specification forwarded to myqueue.

--max-tasks <NUM>

Maximum number of tasks for worker to run.

--tags <TAGS>

Allow worker to pick up tasks with any of TAGS specified as comma-separated list.

--require <TAGS>

Require worker to pick up only tasks with all the TAGS specified as comma-separated list.

--color <WHEN>

Colorize output; use “always” for colors, “never” for no colors, or “auto” (default). Default can be overridden by the TB_COLORS environment variable.

Options:

auto | always | never

--workflow <workflow>

After all work is done, run the workflow given as a parameter (can be a Python import path or a file). Effectively specifying the workflow parameter, worker can generate work for itself infinitely on a dynamical workflow.

--wall-time <wall_time>

Maximum time for worker to run. Worker terminates when walltime is exceeded or would be exceeded within task-wall-time. Default is unlimited time.

Arguments

[WORKER[:NUM]...]

Optional argument(s)

workflow

Run workflow creating folders for tasks inside tree.

SCRIPT is a Path to a Python file with def workflow(runner): function.

Alternatively one can specify a module with -m myrepo.workflow argument.

Usage

tb workflow [OPTIONS] [SCRIPT]

Options

-m, --module <module>

-z, --dry-run

Simulate what would happen, but do nothing.

--silent

Do not print to screen.

Arguments

SCRIPT

Optional argument