https://helpwiki.sharcnet.ca/wiki/index.php?title=Webinar_2021_META:_running_a_large_number_of_jobs_conveniently&feed=atom&action=historyWebinar 2021 META: running a large number of jobs conveniently - Revision history2024-03-29T15:58:33ZRevision history for this page on the wikiMediaWiki 1.36.1https://helpwiki.sharcnet.ca/wiki/index.php?title=Webinar_2021_META:_running_a_large_number_of_jobs_conveniently&diff=490&oldid=prevSyam: Created page with "If you ever had to run lots of small jobs on Compute Canada systems, you likely observed that the process is inefficient, time consuming, and inconvenient. The available tools..."2021-10-07T19:41:54Z<p>Created page with "If you ever had to run lots of small jobs on Compute Canada systems, you likely observed that the process is inefficient, time consuming, and inconvenient. The available tools..."</p>
<p><b>New page</b></p><div>If you ever had to run lots of small jobs on Compute Canada systems, you likely observed that the process is inefficient, time consuming, and inconvenient. The available tools for speeding up the workflow (the built-in Job Array feature of our scheduler, SLURM, GNU Parallel, or GLOST) leave a lot to be desired. Here we introduce our own home-grown package [https://docs.computecanada.ca/wiki/META_package_for_serial_farming META] designed to overcome the shortcomings of the existing tools. META can significantly accelerate your workflow involving running lots of jobs (short or long, serial or parallel, CPU or GPU based). Features like the ability to resubmit all the jobs which either failed or never ran, using a single command, are likely unique to this package. In this webinar we will cover the main features of the package, and will also discuss the typical scenarios when META can become very useful or even essential.</div>Syam