stormtracks Running in Cluster Mode

stormtracks.pyro_cluster.pyro_nameserver – Runs a Pyro4 Nameserver

stormtracks.pyro_cluster.pyro_nameserver.main()

Launches a Pyro4 nameserver

stormtracks.pyro_cluster.pyro_starter – Launches All pyro_workers

stormtracks.pyro_cluster.pyro_starter.main()

Starts all pyro_workers defined in pyro_settings.pyro_workers

ssh’s into each computer and issues a command to start pyro_worker

stormtracks.pyro_cluster.pyro_manager – Manages All pyro_workers

stormtracks.pyro_cluster.pyro_manager.main()

Established communication with and gives tasks to all pyro_workers

N.B. pyro_nameserver must be set up first, and pyro workers must be running on each of the servers defined by pyro_settings.worker servers (either by running pyro_start.py or by manually starting them).

Starts off by finding each of the pyro_workers, then generates a task schedule and uses this to farm out work to each of the workers. Once all work has been done, finished.

stormtracks.pyro_cluster.pyro_ender – Stops All pyro_workers

stormtracks.pyro_cluster.pyro_ender.main()

Kills all pyro_workers defined in pyro_settings.pyro_workers

ssh’s into each computer and issues a command to kill pyro_worker

stormtracks.pyro_cluster.pyro_worker – Processes Data

class stormtracks.pyro_cluster.pyro_worker.PyroWorker

Runs on each of the worker servers

Listens for do_work requests by pyro_manager, on receiving of a test does the required work and saves it to disk using a results manager.

do_work(year, ensemble_member, task)

Do the work required by pyro_manager

Parameters:
  • year – year to analyse
  • ensemble_member – ensemble_member to analyse
  • task – task to do (currently must be ‘vort_track’)
Returns:

dict containing details of task

stormtracks.pyro_cluster.pyro_worker.main()

Sets up a PyroWorker and starts its event loop to wait for calls from a pyro_manager

stormtracks.pyro_cluster.pyro_task – Helps Keep Track of Outstanding Tasks

class stormtracks.pyro_cluster.pyro_task.PyroTask(year, ensemble_member, task)

Represents on task that is to be done

Parameters:
  • year – year of task
  • ensemble_member – ensemble_member of task
  • task – which task to be done
status

Current status of task, must be in STATUSES

task

What task this is doing

class stormtracks.pyro_cluster.pyro_task.PyroTaskSchedule(start_year=2000, end_year=2012, num_ensemble_members=56)

Keeps track of all tasks to be done, and can issue the next outstanding class

Parameters:
  • start_year – year from which to start tasks
  • end_year – year from which to end tasks (inclusive)
  • num_ensemble_members – how many ensemble members to keep tasks for
get_next_outstanding()

Retursn the next outstanding task, None if there are no more

get_progress(years=None, include_year=False)

Returns a string representing the progress for all years

get_progress_for_year(year, include_year=True)

Returns a string representing the progress of the year

  • - :task to be done
  • W :task being worked on
  • + :task complete
  • F :task failure
  • T :task timeout
print_years(years=None, include_year=True)

Prints progress