Diagnostics

Processing records more than its results. As each stage works through an image it also measures a handful of numbers describing how that stage went – how many stars were found, how well the astrometric solution fit, how bright the sky was – and files them in the project’s database against the image and colour channel they came from.

These are worth knowing about for three reasons. They are the quickest way to see whether a night’s data is any good; they are what the pipeline itself consults when it has to choose one image out of many; and when something looks wrong in the final light curves, they are usually where the explanation is.

What gets measured

Each value is stored against one image, one colour channel and one named diagnostic, so every number below exists separately for every channel of every frame.

From calibration

pixel_qNNN

Quantiles of the calibrated pixel values, ignoring masked pixels – pixel_q999 is the 0.999 quantile, and so on. Which quantiles are recorded is configurable, and the names are created as needed, so this is the one family whose members are not fixed in advance. Taken together they describe the brightness distribution of the frame, which is the fastest way to spot cloud, moonlight or a badly wrong exposure.

For example, the median of the pixel values (quantile 0.5) is usually a good measure of the brightness of the sky, since less than half of the pixels in your image should be covered by stars. A perfectly clear sky is darker than clouds, which can reflect light coming from the ground or the moon. Clouds also usually move around, causing the brightness to change over time. So if you see a graph of the median that jumps up and down a lot, that is a good indication that you were not observing in a clear sky.

If you are using a color camera, another thing that can indicate clouds is the color of the sky. Clouds are white, while the sky is blue (even at night). So if you see that the ratio of the median of the red channel to the blue channel is increasing, that can also indicate clouds. The graph below is an example of a night where clouds were present most of the time, but it cleared up near the end.

One thing to keep in mind though is that the brightness and color of the sky is not uniform. The horizon is brighter and could also be less blue than zenith due to light pollution. As you get closer to daylight the brightness can similarly change. As a result, as your telescope tracks the field of sky you are observing it will move further and closer to the horizon, which will also lead to changes in the brightness and color of the sky that can be similar to what clouds do. However, those changes are slow and smooth, unlike clouds. The graph of the median below is an example of observations that happened during a clear night, where dawn is approaching at the end.

From finding stars

num_extracted_src

How many sources the extraction found.

There is no good absolute value to compare against – it depends entirely on your field, your optics and how deep your exposures go. What matters is how it behaves relative to the other frames of the same field. A frame with far fewer stars than its neighbours had something in the way like clouds or stray light drowning out the stars.

A count far above the usual is the more misleading case, because more sources sounds like better data. Often it can mean that the detection threshold has been set low enough to start finding noise, in which case the extra “stars” will not match the catalog – which is what matched_fraction below will tell you.

From plate solving

ra_center, dec_center

Where the middle of the frame actually points, according to the astrometric solution, as opposed to where the mount believed it was pointing.

Plotted against time these trace how the field wandered over a run. A slow crawl in one direction is ordinary tracking error; a sudden step means something moved the telescope, or the mount flipped sides of the pier. Either matters more than it might seem, because your telescope + camera system has different sensitivity to light coming from different directions, and as stars wander around the image their brightness will appear to change. This is what flat field corrections are meant to mitigate – see Bringing your own data for what the calibration frame types are and what doing without them costs you – so this gets even more important if you are not using calibration data.

z_center

The zenith distance of the frame centre – how far from straight overhead the telescope was looking, and so how much atmosphere the light came through. Recorded only when the header says enough about where and when the image was taken.

This is the most useful thing to plot other diagnostics against. Almost everything that degrades with airmass – fainter stars, brighter sky, greater scatter – will show it as a trend with zenith distance, and separating “the target was low in the sky” from “conditions were poor” is a question you will want to answer often.

pointing_offset

How far the frame centre is from the target that was meant to be observed.

Where ra_center and dec_center tell you where you were looking, this tells you how wrong it was, in one number you can sort and threshold.

A jump in the value indicates your mount pointing suddenly shifted (e.g. if you accidentally bumped it). Smooth changes up and down can be caused by imperfect polar alignment, your mount not being powerful enough for the weight of the telescope you have on it, or not well balanced.

A consistently large value from the very first frame usually means the target coordinates or the header keywords they were read from are not what you think.

diagonal_fov

The mean angular distance from the centre of the image to its four corners, as a measure of the field of view that does not depend on the shape of the detector.

For a fixed instrument this should be very nearly constant, which is what makes it useful: it is a check rather than a measurement. A frame whose field of view differs from its neighbours’ has either been refocused or fitted with different optics, or – far more likely – has a plate solution that is quietly wrong.

matched_fraction

The fraction of extracted sources that were matched to catalog stars.

The single most useful indicator that plate solving actually succeeded, as opposed to merely finishing. A solution is rejected outright below min-match-fraction (0.8 by default), so everything you see recorded passed that bar; what is worth attention is a frame sitting just above it, or a run where the fraction is drifting downward.

Read it together with num_extracted_src: if the source count climbed while the matched fraction fell, the extra detections are noise rather than stars.

astrom_residual

The RMS distance, in pixels, between matched sources and the positions the solution predicts for them.

This is how tightly the solution fits. Solutions worse than max-rms-distance (0.5 pixels by default) are rejected, that default being chosen as a little more than the accuracy with which source extraction can place a star’s centre.

A small residual on its own is not proof of a good solution – a handful of stars can always be fit well – so read it alongside matched_fraction. The combination to trust is many matches fitting tightly; the combination to distrust is a tight fit to few.

srcextract_mag_zeropt

The magnitude that corresponds to an extracted flux of one ADU, taken as the median of catalog magnitude plus 2.5 log₁₀(flux) over the matched stars.

It is a measure of how much light is reaching the detector, so it responds to transparency, cloud, dew, and anything else that dims the whole frame at once. Larger means more light: for a fixed star, more flux gives a larger zeropoint. Falling values through a night mean conditions closing in.

Since it is a per-frame number derived before any photometric calibration, it is the earliest warning that a frame is not comparable to its neighbours – available long before magnitude fitting has an opinion.

From fitting star shapes

bg_center

The smoothed background level at the centre of the frame: sky brightness as the photometry actually sees it, after calibration and with the stars taken out. This is usually a higher quality measure of the sky brightness than the median pixel quantile discussed above, because it explicitly excludes the stars.

bg_map_residual

How far the background departs from its own smooth model across the frame.

bg_center says how bright the sky was; this says how evenly. A uniformly bright sky – moonlight, say – raises the first and leaves the second alone. Broken cloud, a passing headlight or stray light entering the optics puts structure into the background that the smooth model cannot follow, and shows up here.

A frame with a modest background but a large residual deserves more suspicion than one with a bright but smooth sky, because uneven background biases stars differently depending on where they sit.

From the source extracted PSF map

s_center, d_center, k_center

The shape parameters of the smoothed PSF/PRF map evaluated at the centre of the frame. s is the one to watch: it measures how concentrated the stars are, so it stands in for focus and seeing. Keep in mind that large s means sharper stars – the light is spread over fewer pixels – so it moves in the opposite direction to the star size you might picture.

Its practical use is as a stand-in for image quality over a run. Falling s means the stars are spreading out: focus drifting as the telescope cools, worsening seeing, or the field sinking toward the horizon. Since it is also the first term in the default rule for picking the photometric reference described below, it is worth knowing which way it points before changing that rule.

d and k describe how far the stars depart from round, and in which direction. d measures elongation along the detector’s x and y axes; k measures it along the two diagonals. Both near zero means round stars, which is what you want.

They have to be read as a pair, because between them they cover every direction: stars trailed along a diagonal show up in k with d near zero, and stars trailed along a detector axis do the reverse. Neither on its own tells you whether the stars are round – it is the two together that do.

The direction is worth attention, because it points at different causes. Elongation that stays put in the same direction frame after frame is usually the instrument: a drive error along one axis, or aberrations in the optics. Elongation whose direction turns steadily through the night is the field rotating, which is what an alt-az mount without a derotator does. Elongation that comes and goes with no pattern is more likely wind or a mount being pushed beyond what it can hold steady.

s_map_residual, d_map_residual, k_map_residual

How much the individual sources scatter about the smooth map of the corresponding parameter.

Where the _center values say what the stars look like in the middle of the frame, these say how well a single smooth model describes the whole frame. A small residual means the star shapes vary gently across the field, as they should; a large one means they vary in ways the map cannot follow.

For s that usually means part of the frame is out of focus relative to the rest – a tilted detector or a curved field. For d and k it means the elongation varies across the frame, which is the signature of optical aberrations growing toward the corners rather than of trailing, since trailing tends to affect the whole frame alike. A residual that is large in all three at once is more often a fit spoiled by bad inputs: saturated stars, blended pairs, or too few sources to constrain the map.

From magnitude fitting

These three are recorded per photometry rather than per image, because each aperture is fit separately, so there is a value for every aperture of every frame.

magfit_residual

The RMS of the correction against the final master photometric reference: how well this frame’s photometry agrees with the ensemble once the fit has done what it can.

This is the closest thing to a single number for “how good is this frame photometrically”, and it is the one to sort by when deciding which frames to look at or discard. Unlike the diagnostics above it is measured after the correction, so it reflects what is left over – the part of the frame’s behaviour that the fit could not explain and therefore could not remove.

Comparing it between apertures is informative in its own right. The aperture with the lowest residual across a run is generally the one to trust for that data, and where the minimum falls tells you something about the size of the stars.

photometry_mag_offset

The best fit offset between the frame’s magnitudes and the reference’s: how much brighter or fainter everything in the frame came out, as a single shift.

This is the bulk correction magnitude fitting applied, so it tracks the same conditions srcextract_mag_zeropt does, but measured after full photometry rather than from extraction, so it should be much more reliable. It is interpreted the same way: an indicator of how transparent the atmosphere was at the time the image was taken.

mag_fit_num_stars

How many stars survived to the last iteration of the fit.

This is the sample size behind the two numbers above, and it is what makes them trustworthy or not. Magnitude fitting rejects outliers as it iterates, so a frame ending on far fewer stars than its neighbours had a lot of its photometry thrown out.

Looking at them

Getting there

The way in is the processing progress page, which is more of a hub than its name suggests: most of what you can see on that table is a link, and hovering over any of them says where it goes.

Each row is a stage, and the three parts of it lead to three different places:

The name of the stage opens its configuration – the settings that stage uses, with the editor already narrowed to them. Convenient, but remember from How a setting gets its value that narrowing the editor does not narrow the effect: many settings are shared, and changing one there changes it for every stage that uses it.

The image type beside it depends on the stage and its state:

• if the stage has recorded errors, it turns red with a warning sign and goes to those errors;

• otherwise, for finding stars, it opens the sandbox for tuning source extraction, where thresholds can be tried against real frames instead of guessed at;

• otherwise, for magnitude fitting, it opens the page for choosing the photometric reference – the frame everything else is calibrated against, and the one place diagnostics are used to make a decision rather than to inform one;

• for every other stage it is plain text and goes nowhere.

Errors take precedence, so a red cell where you expected the tuning sandbox means there are failures to deal with first.

The progress bars, one per image type and channel, open the diagnostics for that stage – “Review astrometry diagnostics”, and so on.

Each stage lands you on the diagnostic that is usually the most telling for it, which is as good a hint as any about which to look at first:

Clicking the progress bar for	opens
calibration	the pixel quantiles
finding stars	num_extracted_src
plate solving	pointing_offset
fitting star shapes	bg_center
aperture photometry	bg_center
the source extracted PSF map	s_center
magnitude fitting, EPD or TFA	the detrending diagnostics

That is only where you arrive. Once on the page you can switch to any other diagnostic, or to a plot of one against another, so the starting point matters less than knowing the door is on the progress page. Stages with nothing to report – those that record no diagnostics – have plain progress bars that do not lead anywhere.

Three kinds of plot

The interface plots them three ways, and which one to reach for depends on the question.

One diagnostic across a run answers “when did this go wrong”: pick a diagnostic and see it for every image in order, with the channels distinguished. A night of cloud, a focus drift, the moon rising – all have shapes you learn to recognise here.

One diagnostic against another answers “why”: scatter any two against each other, and the relationship shows the cause. Residuals rising with zenith distance is airmass; scatter rising with background is moonlight.

Detrending diagnostics work at the level of the light curves rather than the images, showing the scatter left after magnitude fitting, EPD and TFA. That is where you see whether the detrending stages actually improved anything, and by how much, rather than assuming they did.

Every point is a link

A diagnostic is a summary, and a summary can be read wrongly. So the points are not just marks on a plot: clicking one opens the thing it came from, which is how you check that your reading of the plot is right.

On both of the per-image plots – against time and against another diagnostic alike, since they draw the same points the same way – a point takes you to the frame it was measured from. If the median pixel value jumped for three frames in the middle of a run, you do not have to take cloud on trust: click one of them and look. The frame is shown calibrated, as the pipeline saw it rather than as it came off the camera.

This is what makes the diagnostic-against-diagnostic plot more than a correlation. Having found the handful of points that sit away from the trend, you can go straight from each outlier to the frame responsible for it.

That page can do more than show you the image. It will overlay:

• the stars that were extracted, so you can see whether the detections follow real stars or are scattered over noise;

• the stars that were projected there from the catalog, and those that were matched between the two, which together turn matched_fraction from a number into a picture – a bad plate solution looks like two sets of points that do not line up;

• the pixels below the background, and the pixels below each recorded quantile, colouring the frame by brightness. This is the companion to the pixel_qNNN diagnostics: where those tell you the sky got brighter, this shows you whether it did so evenly, as moonlight does, or in patches, as cloud does.

On the detrending plots, where each point is a star rather than a frame, clicking goes to that star’s light curve instead – which is the natural question there, since a point sitting above the crowd is asking to be looked at.

What the pipeline does with them

Diagnostics are not only for looking at. Choosing the single photometric reference – the frame every other frame’s photometry is calibrated against – is decided by scoring the candidates on their diagnostics.

The score is an expression you can change, and it can use any diagnostic in two derived forms: qnt_<name> is the frame’s rank among the candidates for that diagnostic, from 0 to 1, and std_<name> is the spread of that diagnostic across them. Ranks are usually what you want, since they do not care what units the diagnostic is in. The default is:

1.0 / ((1.0 - qnt_s_center)**2 + qnt_bg_center**2)

which reads, once unpacked, as prefer the frame with the smallest stars and the darkest sky – the sharpest image on the least contaminated background, which is what makes a reference other frames can be measured against.

Because the ranking is only over the candidates in hand, a reference is always the best of what was actually observed. It is worth looking at the diagnostics of the frame that gets chosen: if the best available frame is poor, everything calibrated against it inherits that.