Webinar 2021 META: running a large number of jobs conveniently

If you ever had to run lots of small jobs on Compute Canada systems, you likely observed that the process is inefficient, time consuming, and inconvenient. The available tools for speeding up the workflow (the built-in Job Array feature of our scheduler, SLURM, GNU Parallel, or GLOST) leave a lot to be desired. Here we introduce our own home-grown package META designed to overcome the shortcomings of the existing tools. META can significantly accelerate your workflow involving running lots of jobs (short or long, serial or parallel, CPU or GPU based). Features like the ability to resubmit all the jobs which either failed or never ran, using a single command, are likely unique to this package. In this webinar we will cover the main features of the package, and will also discuss the typical scenarios when META can become very useful or even essential.