tpar

simple, parallel job scheduling http://github.com/bgamari/tpar/

Latest on Hackage:0.1.0.0

This package is not currently in any snapshots. If you're interested in using it, we recommend adding it to Stackage Nightly. Doing so will make builds more reliable, and allow stackage.org to host generated Haddocks.

BSD3 licensed by Ben Gamari
Maintained by bgamari@gmail.com
# tpar — simple parallel job scheduling

`tpar` is a simple tool for concurrent job scheduling. Say you have a
directory full of files which need processing,

```bash
$ ls
file1 file2 file3 file4 file5
...
```

Usually one could use a `bash` for loop,

```bash
$ for f in *; do process $f; done;
```

But if `process` is a long-running task and you have many cores at
your disposal, it would be nice to speed things up a bit,

```bash
$ tpar server -N8
$ for f in *; do tpar enqueue -- process $f; done;
```

If you have multiple machines with the data mounted over, say, NFS, they can
also help with churning through the queue,

```bash
$ for m in worker1 worker2 worker3; do
> ssh $m -- tpar worker -H`hostname`;
> done
```

## Commands

`tpar` has several subcommands,

* `tpar server` starts a local queue server.
* `tpar worker` starts a worker associated with
the given queue
* `tpar enqueue -- $cmd` enqueues a job in the given queue
* `tpar status` allows you to query for the status of the queue. You can also provide a job match expression
* `tpar kill` kills a running task (specified by a job match expression)
* `tpar watch` is analogous `tail -f`, watching the output of a set of running tasks
* `tpar dump` dumps a JSON representation of the queue state.

Nearly all of these commands will require that the `-H` option be provided
specifying the canonical hostname of the queue server (the machine running
`tpar server`).


## Job match expressions

Several `tpar` commands accept a *job match expression*, which specifies the
subset of jobs on which the command should act. For instance (note the quotes to
ensure that `bash` doesn't interpret our symbols),

```
$ tpar status '*' # This is equivalent to `tpar status` run without an argument
$ tpar status id=2
$ tpar status state=running
$ tpar status 'name="my-job" or name="my-other-job"'
```

These expressions consist of the primitive matches,
* `name="STRING"`, which matches on the job name provided in the `--name` of
`tpar enqueue`.
* `id=`, which matches on the job ID
* `state=`, which matches on the current state of the job (`queued`, `running`,
`finished`, `failed`, `killed`, or `code=N`)
* `*`, which matches all jobs

These matches can be connected with the `and` and `or` operators, and inverted
with `!`.
comments powered byDisqus