When something goes wrong

A stage that fails does not simply print a complaint and vanish. The failure is recorded in the project, along with enough of its surroundings to work out afterwards what happened – and, if that is not enough, to hand the whole situation to someone else.

The error log

A red button appears in the interface’s navigation as soon as there is anything to report, and is absent when there is not, so there is no need to go looking. It leads to the list of recorded failures.

Opening one shows what went wrong: the message, the details gathered at the moment of failure – which stage was running, on which image, and the files it was working with – and the traceback. Once a failure has been dealt with it can be marked resolved so it stops drawing attention, or deleted outright.

Not every failure stops everything. A run that comes to grief part way through has still recorded everything it finished, so setting processing going again carries on from where it stopped rather than starting over.

Picking up after an interrupted run

Carrying on is not merely a matter of skipping what was finished. A run that stops without warning – a crash, a power cut, a job killed by hand – leaves whatever it was in the middle of half done, and a half-written result is worse than none, because nothing downstream can tell it apart from a complete one.

Rather than trying to undo everything centrally, each stage is responsible for its own leavings. A stage says which states it can be caught in and which it is able to resume from, and the ones that write results in pieces provide a way to undo a partial attempt. When processing next starts, anything recorded as begun but not finished is handed back to the stage that began it, which clears up and reports the state it has left things in; that state has to be one the stage can resume from, and the pipeline refuses it otherwise. Aperture photometry, for instance, deletes the measurements it had started writing into the data reduction file, so the next attempt begins from a clean one.

Most stages are simple in this respect: they record only that they had begun, so there is one state to be caught in and clearing up is straightforward. Building light curves is the exception worth knowing about, since it adds its points first and marks the files complete afterwards, and so can be interrupted between the two – in which case the work is not thrown away, only finished off.

None of this needs anything from you. It happens as processing starts, and the only sign is a warning in the log saying what was cleaned up.

The logs

Every process writes two files into logs/ under the project home: the log proper, and a second holding whatever the process wrote to its output directly. Their names are built from the stage, the task, the time and the process id, so a run leaves a separate trail for each stage rather than one tangled file. logging-fname and std-out-err-fname control the naming, and verbose decides how much detail goes in.

Reading these is worthwhile when a stage produced something odd rather than failing outright, which is the case the error log cannot help with.

Sending a crash report

If a failure is not something you can act on, the useful thing to send is a crash report: one zip file holding the failure and everything around it. The button is on the error’s own page, labelled Download crash report. The same thing from a terminal:

wisp-crash-report /path/to/project/home --last

Name an error explicitly instead of --last to report on an older one; the number is the one the error log shows.

The zip holds the error record, the details sidecar with the full traceback, the logs belonging to that run, a copy of the project database, and a note of the platform, Python and package versions in use. A manifest.json lists what was gathered. The database copy is what makes the report worth sending: it carries the configuration in force, what had been processed already, and the equipment involved, which is usually the difference between a guess and an answer.

Credentials are removed. Scrubbing is not optional and nothing is written unscrubbed: values whose names mark them as secret – the Gaia archive login, the astrometry.net API key, anything called a password or a token – are replaced with ***REDACTED*** wherever they appear, in the logs, in the configuration and in the copied database alike.

Collection is deliberately best-effort. A file that cannot be read, or that is not there, is written down as a gap in the manifest rather than being allowed to abandon the report – so a report still arrives in the case where the missing file is itself the problem. Logs are truncated to the beginning and end of each file, half a megabyte in all; pass --max-log-bytes for more if it is asked for.

Keeping the log from growing

Recorded failures stay until removed. To prune them:

wisp-cleanup-errors /path/to/project/home --older-than 30d

The cutoff takes days, hours or weeks (30d, 12h, 2w). Run without it, the command leaves the records alone and only tidies up after itself, clearing sidecar files with no error still pointing at them and references to sidecars that have gone.