Webinar 2025 Serial Farms: Package options and when to switch to farming

Small jobs are convenient to submit individually, but at scale they can overload a scheduler, inflate queue times, and ultimately reduce throughput. This webinar examines practical strategies for consolidating large numbers of short tasks, including job arrays, task bundling techniques, and wrapper-based aggregation. We will discuss how these approaches differ in overhead, portability, scheduler behavior, and job failure handling. The session will also provide guidance on recognizing when packaging options no longer yield sufficient throughput and when transitioning to a serial farming model becomes advantageous. Serial farms can mitigate scheduler pressure, improve wait times on busy clusters, and offer more predictable performance. Examples will be provided throughout the Webinar and shared on GitHub for future reference.