Metadata-Version: 2.4
Name: seamm_jobserver
Version: 2026.8.10
Summary: The JobServer for the SEAMM environment.
Home-page: https://github.com/molssi-seam/seamm_jobserver
Author: Paul Saxe
Author-email: psaxe@molssi.org
License: LGPLv3+
Keywords: seamm_jobserver
Platform: Linux
Platform: Mac OS-X
Platform: Unix
Platform: Windows
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Chemistry
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: AUTHORS.rst
Requires-Dist: seamm
Requires-Dist: seamm_slurm
Requires-Dist: psutil
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: platform
Dynamic: requires-dist
Dynamic: summary

===============
SEAMM JobServer
===============


.. image:: https://img.shields.io/github/issues-pr-raw/molssi-seamm/seamm_jobserver
   :target: https://github.com/molssi-seamm/seamm_jobserver/pulls
   :alt: GitHub pull requests

.. image:: https://github.com/molssi-seamm/seamm_jobserver/workflows/CI/badge.svg
   :target: https://github.com/molssi-seamm/seamm_jobserver/actions
   :alt: Build Status

.. image:: https://codecov.io/gh/molssi-seamm/seamm_jobserver/branch/main/graph/badge.svg
   :target: https://codecov.io/gh/molssi-seamm/seamm_jobserver
   :alt: Code Coverage

.. image:: https://github.com/molssi-seamm/seamm_jobserver/workflows/CodeQL/badge.svg
   :target: https://github.com/molssi-seamm/seamm_jobserver/security/code-scanning
   :alt: Code Quality

.. image:: https://github.com/molssi-seamm/seamm_jobserver/workflows/Release/badge.svg
   :target: https://molssi-seamm.github.io/seamm_jobserver/index.html
   :alt: Documentation Status

.. image:: https://img.shields.io/pypi/v/seamm_jobserver.svg
   :target: https://pypi.python.org/pypi/seamm_jobserver
   :alt: PyPi VERSION


The JobServer for the SEAMM environment


* Free software: BSD license
* Documentation: https://molssi-seamm.github.io/seamm_jobserver/index.html
* Code: https://github.com/molssi-seamm/seamm_jobserver


Features
--------

The SEAMM JobServer handles executing Jobs for the Dashboard, either as
local subprocesses (the default) or, if configured, by submitting each job
as a SLURM batch job -- see the User Guide.


Acknowledgements
----------------

This package was created with the `molssi-seamm/cookiecutter-seamm-plugin`_ tool, which
is based on the excellent Cookiecutter_.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`molssi-seamm/cookiecutter-seamm-plugin`: https://github.com/molssi-seamm/cookiecutter-seamm-plugin

Developed by the Molecular Sciences Software Institute (MolSSI_), which receives funding
from the `National Science Foundation`_ under award CHE-2136142.

.. _MolSSI: https://molssi.org
.. _`National Science Foundation`: https://www.nsf.gov


=======
History
=======
2026.8.10 -- Route jobs to multiple queues/clusters from one JobServer instance
    * A JobServer instance can now be configured with more than one queue
      (cluster/section) in its ``<root>/<jobserver-name>.ini`` and route
      each job to the one it asks for (``parameters["queue"]``), falling
      back to the instance's default queue if it doesn't specify one.
      Requesting an unknown queue, or no queue with no default configured,
      fails the job immediately (``startup error``) rather than silently
      running it somewhere unintended.
    * Each queue's ``max_concurrent_jobs`` is enforced independently, so
      one full queue never blocks another from accepting new jobs, and
      SLURM polling is batched per queue so two different clusters'
      same-numbered SLURM job ids are never compared against each other.
    * A ``type = local`` queue (see ``seamm_slurm``'s ``2026.8.10``
      release) and one or more ``type = slurm`` queues can now coexist on
      the same JobServer instance -- some jobs run as local subprocesses,
      others dispatch to real SLURM clusters, from one process.
    * A SLURM job's ``job_data.json`` now records which queue it ran on
      and its SLURM job id (found from real usage: nothing previously told
      a user which cluster a finished job had actually run on).
    * No ``<root>/<jobserver-name>.ini`` at all means behavior is
      unchanged from before this feature existed -- every job still runs
      as an uncapped local subprocess.

2026.8.9 -- Bugfix: startup reattachment could misapply SLURM staging to a local job
    * On restart, a job's ``running``/``kill`` datastore row was reattached
      based on whether the JobServer instance is *currently*
      SLURM-configured, not on what that specific job actually is. A local
      job predating SLURM being enabled on this instance (or, worse, one
      still genuinely running) could be treated as a lost SLURM submission:
      at best retrying a pointless remote file transfer forever instead of
      reading its already-correct local status, at worst getting resubmitted
      via SLURM as a real duplicate run of a job still running locally.
      Reattachment now checks each job's own recorded pid or SLURM job id,
      not the instance's current mode, so a local job is always recognized
      as local regardless of whether SLURM is configured.
    * Bugfix: a locally-run job resumed after a restart could raise an
      error the next time it finished, from a missing internal field.
    * A locally-run job whose process ended with no other evidence of its
      outcome now has its actual result read from the job's own record
      rather than being assumed successful.

2026.8.8 -- Support SLURM dispatch to a cluster with no shared filesystem
    * A JobServer can now dispatch jobs via SLURM to a cluster it shares no
      filesystem with -- for example, a laptop reaching a remote cluster
      over ssh. A job's working directory is staged there before
      submission and its results staged back after, automatically.
      Configure it with new ``remote_root`` and
      ``remote_run_from_jobserver`` (or ``remote_conda_env``) options in
      the relevant section of ``<root>/<jobserver-name>.ini``. See the
      User Guide for the full option list.
    * Internal: the JobServer itself, not the running job, now writes a
      job's final status to the datastore, for every job type (this was
      already true for the SLURM-losing-track case; now it's the normal
      path for local jobs too). Fixes a real gap: a local job that
      crashed hard before it could write its own status previously stayed
      stuck as ``running`` forever, with no recovery -- the JobServer now
      finalizes it from the process's exit status instead.
    * Live-validated end to end against a real cluster: a real job staged
      out via rsync, submitted via sbatch, confirmed completed via SLURM's
      own accounting, and staged back correctly.

2026.8.7 -- Actively stop deleted or explicitly-killed jobs
    * Deleting a job (e.g. via the dashboard) removes its row and files, but
      previously left whatever was actually running it untouched -- it just
      ran on until it crashed on its own missing files. The JobServer now
      notices a tracked job's row disappearing and actively stops it
      (``scancel`` in SLURM mode, terminating the process in local mode),
      every poll cycle.
    * A job can also be stopped without deleting it or its files, by setting
      its ``status`` to ``kill`` -- an ordinary status update, no new API.
      The JobServer stops the run the same way, then sets ``status`` to
      ``killed``. A job killed before it was ever started is finalized as
      ``killed`` directly.
    * Both checks are also applied during startup reattachment, so a kill
      requested just before a JobServer restart is not lost or, worse,
      mistaken for a merely-lost job and resubmitted.

2026.8.6 -- Add SLURM submission mode (Option 1: whole-flowchart sbatch)
    * A JobServer instance can now submit each job as a single SLURM batch job
      instead of a local subprocess, controlled by a new, opt-in
      ``<root>/<jobserver-name>.ini`` config file (absent means unchanged,
      local-subprocess behavior). Supports both running the SLURM CLI directly
      on the JobServer's own host and reaching a remote cluster over SSH.
    * Added a configurable cap on how many jobs a JobServer instance keeps
      outstanding in SLURM at once.
    * If a tracked job's SLURM state goes missing or terminal while its
      datastore row still shows ``running`` (a node failure, an OOM kill, a
      cancellation, or the JobServer itself having restarted), the JobServer
      now trusts the job's own recorded outcome when available, otherwise
      safely resubmits it (flowcharts checkpoint and resume, so this does not
      redo completed work) up to a configurable retry cap before giving up.
    * A job can now request different SLURM resources (cores, memory,
      walltime, ...) than the JobServer instance's own defaults, via
      ``parameters["slurm"]`` on the job row -- gated by a new, optional
      ``[<section>.limits]`` ini section that says which fields a job may
      override and within what bounds (enumerated choices or numeric/size/
      time ranges). Secure by default: no ``.limits`` section means nothing
      is overridable. Always re-validated server-side regardless of what
      submitted the job.
    * See the User Guide for the config file format and behavior.

2025.11.22 -- Bugfix: Catching errors when starting a job.
    * Now catch any error that occurs starting a job, and set its status to 'startup
      error'. This prevents the JobServer getting into a loop repeatedly trying to
      submit the job.

2025.11.12 -- Added SEAMM_JOBSERVER and SEAMM_JOB_ID environment variables
    This PR adds support for passing job-specific metadata to spawned processes through
    environment variables. Jobs can now access their unique job ID and the name of the
    JobServer that spawned them.

    * Added SEAMM_JOB_ID and SEAMM_JOBSERVER environment variables for spawned job
      processes 
    * Added --name command-line argument to specify JobServer name (defaults to
      hostname)
    * Cleaned up docstring formatting

2024.4.12 -- Fixed issue with status of finished jobs
   * Fixed a problem if a job returned a status of None, which was reported as an
     error.

2024.4.11 -- Correcting description of this package

2024.4.5 -- Adding support for debugging
   * Use the value of the environment variable SEAMM_LOG_LEVEL to set the log level for
     jobs. DEBUG, INFO, WARNING are three useful levels.
     
2024.1.17 -- Changes to support running in Docker containers.

2023.12.12 -- Improved the output in the GUI.
   * Improved the output to the GUI
   * Fixed a bug in the file path for the status file.

2023.3.23 -- Substantial improvements to JobServer
   * Switched to independent process for Jobs, which means they are fully independent of
     the JobServer and continue to run if the JobServer stops
   * Discover existing running jobs on startup and monitor them.
   * Added status information for the machine the JobServer is on as well as Jobs
   * Provide a GUI if run from the commandline, showing the log and status.

0.9.1 (2020-05-29)
------------------

* First release on PyPI.
