How py-cluster-api stacks up against every in-house programmatic bsub/bjobs/bkill wrapper found across the JaneliaSciComp and janelia-cellmap GitHub orgs.
extension:py bsub) was run across both orgs, returning 59 file hits in 30 repositories. Each hit was read and classified as either a reusable submission library (a function/class others can import to submit, poll, or cancel LSF jobs generically) or a one-off script (a hardcoded pipeline step that happens to shell out to bsub once). Six files across four repos qualified as libraries and are compared in full below; the rest — 19 one-off scripts, plus several false positives where "bsub" was incidental — are listed in the appendix for transparency. fileglancer was also found using bsub, but it turned out to be a downstream consumer of py-cluster-api itself (a pinned dependency), so it's discussed separately rather than scored as a competing implementation.
| Feature | py-cluster-api this repo — reference |
tpt / fuster.py
JaneliaSciComp — bqueue_type
|
dacapo / compute_context
janelia-cellmap — Bsub(ComputeContext)
|
cellmap-flow / bsub_utils janelia-cellmap |
dask-janelia / deploy.py
JaneliaSciComp — thin dask_jobqueue wrapper
|
tensorswitch / batch.py JaneliaSciComp | transfero JaneliaSciComp |
|---|---|---|---|---|---|---|---|
| Submission & scripting | |||||||
| Async API | ✓ | ✗ | ✗ | ✗ThreadPoolExecutor for concurrency, not asyncio | ✗ | ✗ | ✗ |
| Structured resource spec (cpus/gpus/mem/walltime/queue) | ✓ | ✗only slot_count; rest is a raw CLI-flag list | ◐queue/gpus/cpus/billing fields; no memory or walltime | ◐queue/charge_group/gpus/cpus; no memory or walltime | ◐threads/walltime explicit, rest is untyped kwargs | ◐memory_gb/wall_time/cores + auto-calc helper | ✗only slots_per_job |
Job script templating (writes a #BSUB directive file) |
✓ | ✗calls bsub directly with CLI args | ✗direct bsub CLI-arg invocation, no script file | ✗inline command string via bsub, no script file | ✗delegated entirely to dask_jobqueue | ◐writes a bash script, but it calls bsub -K itself, not #BSUB pragmas |
✗ |
| Prologue / epilogue hooks | ✓ | ✗ | ✗ | ✗ | ◐hardcoded thread-limiting env exports only, no epilogue | ✗ | ✗ |
Array job submission (-J "name[1-N]") |
✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗loops individual submissions instead |
| Per-element array status tracking | ✓ | ✗ | ✗ | ✗ | ✗ | ◐inferred from output-file existence, not bjobs per-index | ✗ |
Job dependency chaining (-w 'done(...)' / sequential bwait) |
✗gap — see below | ✗ | ✗ | ✗ | ✗ | ◐coordinator script chains steps with sequential bsub -K/bwait |
✗ |
| Monitoring & lifecycle | |||||||
Status polling (bjobs) |
✓bjobs -json | ◐plain-text bjobs, batched 10k IDs at a time | ✗bjobs never called anywhere in the repo | ✓bjobs -noheader, manual column parsing | ✗worker phone-home model instead of polling | ✓bjobs -noheader -o stat | ✗delegated to external tpt.fuster |
| Active background polling loop (monitors all tracked jobs on an interval) | ✓ | ◐synchronous blocking loop, not a background monitor | ✗ | ◐on-demand, single-job polling loop | ✗ | ✗status checked only on-demand | ✗ |
| Callback/event dispatch on completion | ✓on_success / on_failure / on_exit | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Blocking wait for job(s) | ✓monitor.wait_for() | ✓bwait() / bqueue.run() | ✗only waits for the bsub call to return, not job completion | ◐waits for a scraped output string, not job status | ✗ | ✓bsub -K, bwait -w | ✓delegated to tpt.fuster's bqueue.run() |
Cancel single job (bkill) |
✓ | ✗ | ✗ | ✓ | ◐bkill -d command override, no clean cancel API | ✗ | ✗ |
| Cancel all tracked jobs | ✓ | ✗ | ✗ | ◐SIGINT/SIGTERM cleanup handler only, no public API | ✗ | ✗ | ✗ |
| Cancel by name/pattern | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Reconnect / rediscover jobs after restart | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗lock file just prevents concurrent runs |
| Adopt an existing job ID without resubmitting | ✓track() | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Zombie / stale job detection | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Configuration & architecture | |||||||
| Config file with named profiles | ✓Nextflow-style YAML profiles | ✗ | ◐one active compute_context block, not switchable named profiles | ◐app config YAML, not scheduler resource profiles | ✗just a deployment string switch | ✗ | ◐one YAML file per user, not named profiles |
| Env control (inherit env vars / login shell) | ✓inherit_env, login_shell | ✗ | ✗ | ✗ | ◐forces single-threaded BLAS/OMP vars only | ✗ | ✗ |
| Multi-scheduler abstraction (pluggable backends, not LSF-only) | ✓abstract Executor; LSF + Local implemented | ✗ | ✓ComputeContext ABC — LocalTorch + Bsub backends | ◐Job ABC exists; AWS/GCP/Azure/Slurm subclasses are unused demo code | ◐if/else LSF-vs-Local dispatcher, no formal ABC | ✗ | ✗ |
| Local execution fallback (run without a real scheduler) | ✓dedicated LocalExecutor | ✓do_actually_submit=False sentinel mode | ✓LocalTorch backend | ✓LocalJob backend | ✓auto-detects via bsub_available(), falls back to LocalCluster | ✗ | ✗ |
Log file path templating with job ID (%J/%I) |
✓ | ✗single fixed path, no job-ID placeholder | ✗job ID unknown at submit time (captured after bsub returns) | ✗ | ✗ | ✓ | ✗ |
| Memory unit parsing/conversion (e.g. "16GB" → LSF units) | ✓ | ✗no memory param at all | ✗no memory field at all | ✗ | ✗memory string passed through unparsed | ✓GB→MB conversion | ✗ |
| Job name prefixing/namespacing | ✓ | ✗ | ✗job name hardcoded to "dacapo" | ◐sanitizes names, no prefix filter | ✗ | ✓ | ✓ |
| Client-side concurrency throttle for independent (non-array) jobs | ✗gap — see below | ✓slot-limited bin-packing scheduler | ✗ | ✗ | ✗ | ◐array-level %N throttle only, same mechanism py-cluster-api has | ✗ |
| CLI interface | ✗library only, by design | ✗ | ✓click-based dacapo CLI (not job-submission specific) | ✓ | ✗ | ✓ | ✓includes a cron-installer CLI |
Across every implementation surveyed, three ideas turned up that py-cluster-api doesn't currently cover:
tensorswitch's "coordinator script" runs several bsub -K steps back-to-back so each stage waits for the previous one (image → pyramid → labels → pyramid), and an older qsub-migrated script (twocof) used explicit -w 'ended(...)' conditions. py-cluster-api has no depends_on/-w equivalent today — pipelines have to poll and re-submit manually.tpt/fuster.py's bqueue_type queues arbitrary jobs with a declared "slot cost" and only submits as many as fit under a running-slot cap, backfilling as jobs finish. py-cluster-api only throttles concurrency within a single array job via LSF's own %N syntax — there's no equivalent for a batch of otherwise-unrelated single jobs.py-cluster-api is deliberately library-only. Likely fine as-is, but worth naming as a deliberate scope choice rather than an oversight.py-cluster-api is a strict superset of every reusable LSF-submission library found across both orgs. Nothing else surveyed combines async submission, structured resources, array-job element tracking, JSON-based active polling, callbacks, cancellation (by ID, by name, or all), reconnect-after-restart, or YAML profiles — most implementations have at most 2-3 of those, and the rest hand off the hard parts (polling, waiting, cancellation) to something else entirely, whether that's dask_jobqueue, an external in-house module, or nothing at all.
JaneliaSciComp/fileglancer
also matched the "bsub" search, but it isn't an independent implementation — its pyproject.toml pins
py-cluster-api >=0.7.0,<0.8 as a dependency, and
fileglancer/apps/jobs.py imports ResourceSpec directly and dispatches
submit/poll/cancel/reconnect calls straight through py-cluster-api's executor. It's included here
as a validation point rather than a matrix row: a real production app is exercising
inherit_env/login_shell, job_name_prefix-based reconnect(),
and array-free single-job submission, and layering its own DB-backed job table, per-user privilege
separation (root-owned server dispatching through unprivileged per-user workers to satisfy root-squash),
manifest-driven prologue/epilogue, and a second zombie-detection pass on top. None of that duplicates
py-cluster-api's own bsub/bjobs/bkill mechanics — it's exactly the kind of app-layer logic the
library is meant to be built on top of.
The rest of the 59 code-search hits were read and classified as one-off pipeline scripts (a hardcoded job tied to one specific tool, not something another script could import and reuse) or false positives ("bsub" appearing incidentally — in a comment, a help string, or an unrelated substring match). Listed here for completeness.
| Repo | File | Why excluded |
|---|---|---|
| JaneliaSciComp/dlc-trainer | dlc_trainer.py | Hardcoded singularity/GPU training job for one model |
| JaneliaSciComp/dilucid | dilucid.py | Hardcoded singularity/GPU DLC-inference job |
| JaneliaSciComp/ssh_cluster_tunnel_for_vscode | tunnel.py | Single hardcoded self-resubmitting tunnel job |
| JaneliaSciComp/delectable | compress_videos_on_cluster.py | Hardcoded per-file video-compression job |
| JaneliaSciComp/twocof | process0cluster.py | Old qsub-migrated pipeline script, hardcoded paths/names |
| JaneliaSciComp/fvb | exercise_pipeline.py | Consumer of tpt/fuster.py, not its own implementation |
| JaneliaSciComp/ExtractionAndNavigationalAnalysis | settings/functions.py | Old qsub-migrated batching helper, tightly coupled to this pipeline's MATLAB paths |
| JaneliaSciComp/neuronbridge-precompute | bin/create_ppp_sync_submitter.py, bin/deprecated/copy_ppp_imagery.py | Writes bsub text into a shell script; never invokes it directly in Python |
| JaneliaSciComp/Cloud_Uploader, hortacloud-importer | cloud_uploader.py, tiff2octree.py | Near-identical copy-pasted dask_jobqueue.LSFCluster wrappers (same pattern as dask-janelia, above) |
| JaneliaSciComp/SOFIMA-scripts | 2-planes-warp.py, multiscale.py, 2-planes-flow-mesh.py | "bsub" appears only in a comment telling the user to launch an interactive shell first |
| JaneliaSciComp/mouse-side-jaw-nose-tongue | mouse-side-jaw-nose-tongue-one.py | Commented-out dead code, never executed |
| JaneliaSciComp/VisibleBurrowSystem | ROS/ephysProc/src/py_pubsub/setup.py | False positive — incidental substring match |
| JaneliaSciComp/contactome_matching | scripts/run_matching_pair.py | "bsub" only in a docstring example |
| JaneliaSciComp/fileglancer | fileglancer/cli.py | Mention only in a debug-command docstring; actual logic is in jobs.py (discussed above) |
| janelia-cellmap/cellmap-segmentation-challenge | utils/batch_eval.py, cli/evaluate.py | Templated command + log-tail polling, but hardcoded to one CLI command |
| janelia-cellmap/process-blockwise | predict.py, predict_daisy.py | YAML-driven but hardcoded to this predict pipeline's args |
| janelia-cellmap/ask-to-mask | scripts/submit_lora_lsf.py | Well-written argparse resource spec, but tied to one Flux LoRA training command |
| janelia-cellmap/exp_mito | runs/submit.sh.py, generate_val.py, generate_val_all.py | Notebook-style, hardcoded paths/setup lists |
| janelia-cellmap/annotator-metrics | monitor.py | os.system("bsub ...") with hardcoded paths/emails |
| janelia-cellmap/utils-share | rhoadesj & zouinkhim daisy/postprocessing scripts (4 files, 2 of them byte-identical duplicates) | Hardcoded daisy-worker submission scripts |
| janelia-cellmap/segmentation-challenge-meshes | faileds_cluster_submission.py, generic/cluster_submission.py (98% duplicate of each other) | Hardcoded bsub ... meshify string + polling throttle |
| janelia-cellmap/annotator-metrics, cellmap-analyze, igneous-daskified | io_util.py (shared lineage across all three, traced to flyemflows) | Shared logging/argparse/timing utility; "bsub" appears only in a --num-workers help string, no actual submission code |
| janelia-cellmap/exp_salivary | postprocess/scripts/check_eval_size.py | "bsub" only in a trailing comment showing manual invocation |