Skip to content

Scheduled Jobs

Scheduled jobs have three components:

  • jobs - the top-level operation that defines when a job is executed by calling a service
  • services - the code for the job
  • scheduler - the long-running process that executes jobs according to their scheduling configuration

Jobs

The jobs.py file is intended as a shorthand index of available jobs. Each top-level job in jobs.py should contain as little code as possible, ideally just calling a single service function.

Each job should have its own configuration in the config object that allows it to be enabled/disabled and otherwise have its scheduling configured.

Services

Services should be relatively self-contained, such that disabling a service does not cause the site to become unusable.

Services should be async and make careful use of resources, such as limiting the number of concurrent async calls with a Semaphore or otherwise. Avoid long-running processes, and split jobs into small, atomic units when possible so that failure of any part of them doesn't break the whole thing.

Services should not have side effects except for mutating the database.

Services should export a single function with no parameters up to sciop.services to be called from jobs.py

Scheduler

The scheduler module wraps APScheduler with a handful of convenience methods. It is started and torn down with starlette's lifespan contextmanager methods.

The scheduler uses the database as its job store, so jobs can be persisted in the case of server crashes, though currently they are removed when the server is requested to do an orderly shutdown.