Developer API
This page summarizes the template-building code in developer/.
These routines are intended for constructing, inspecting, and validating
wisp templates. This is intended for advanced users who need to build wisp templates using their own data.
Overview
The developer notebook workflow is 1. preprocess.ipynb
2. nmf.ipynb.
preprocess.ipynb
prepares the data for NMF fitting, including applying the flat field, masking sources, masking bad pixels,
and identifying the wisp region.
nmf.ipynb builds the
wisp multi-component template using the NMF algorithm.
Notebooks
| Notebook | Purpose |
|---|---|
preprocess.ipynb |
Preprocesses the data by applying the flat field, masking sources, masking bad pixels, and identifying the wisp region. |
nmf.ipynb |
Builds the NMF wisp template. |
nmf_withflat.ipynb |
(Experimental) Wisp fits joint with flat field fitting. |
Core Code Modules
| Module | Role | Main outputs |
|---|---|---|
preprocess.py |
Background subtraction, masks, 1/f correction, wisp-region detection. | flat and 1/f-corrected data, source masks, wisp region. |
nmf.py |
Core weighted NMF fitting on vectorized images. | NMF templates and their amplitudes in each exposure. |
nmf_beta.py |
(Experimental) NMF variants with iterative 1/f correction and flat fitting. | NMF templates, amplitudes in each exposure, and best-fit flat field. |
postprocess.py |
Residual-based bias and uncertainty estimation. | Wisp bias and wisp uncertainty maps. |
utils.py |
FITS loading, CRDS flat handling, plotting. | Loaded arrays, quick-look figures, outlier identification. |
Preprocessing
estimate_background(data, mask)
Estimates one scalar background per exposure from unmasked finite pixels using sigma-clipped statistics.
Returns bg with shape (n_exp,).
calibrate_rates(rates, error, flat, mask, wisp_mask)
Applies flat-field calibration, estimates the background outside the science and wisp masks, and returns a background-subtracted stack.
Returns bsub, cal_e, bg.
correct_1f_noise(data, mask, wisp_mask=None, split_amplifier=False, beta_version=True)
Estimates horizontal and vertical stripe structure and subtracts it from a 3D stack. When a wisp mask is supplied, it is dilated before fitting the stripes.
Returns the stripe-corrected stack.
mask_bad_pixels(data, err, mask, std_threshold=10)
Builds an additional hot- and cold-pixel mask from exposure-level and stack-level deviations.
Returns a boolean mask with bad pixels marked True.
build_wisp_regions(data, edges=None, max_comp_num=1, ...)
Interpolates missing pixels, detects segmented structures in a stacked image, selects the brightest candidate regions, dilates them, and optionally excludes nearby source regions.
Returns wmask, segment_map.
NMF Fitting
nmf_basic(data, invVar, A, W, n_iter=100, verbose=False)
Runs multiplicative weighted NMF updates for fixed initial
amplitudes A and components W. Inputs are
vectorized 2D arrays with shape (n_exp, n_pix).
Returns normalized A, W.
nmf_iterative(data, invVar, n_comp, n_iter=100, init_amp_ratio=0.1, outliers=None, verbose=False)
Adds NMF components one at a time, refits all components after each addition, and solves amplitudes for excluded outlier exposures with non-negative least squares.
Returns A_all, W.
vectorize_data(data, err, mask, valid_region)
Selects pixels that contain finite data and belong to the wisp region, flattens each exposure, converts errors to inverse variance, and zeroes masked or NaN pixels.
Returns data, invVar, selected_pixels.
transform2D(data, selected_pixels, fill_value=0)
Restores vectorized 1D data or component stacks to the original 2D image geometry.
identify_outliers(A, std_threshold=8)
Uses sigma-clipped component amplitudes to flag exposures with unusually large NMF coefficients.
Returns sorted unique exposure indices.
nmf_beta.py variants
nmf_iterative_correct1f alternates NMF fitting with
beta 1/f stripe estimation. nmf_iterative_fit_flat
adds a flat component alongside the wisp components. These paths
are useful for template experiments and should be validated against
the standard nmf.py workflow before adoption.
Postprocessing
estimate_errors(residual, invVar, outliers, selected_pixels, nan_pixels)
Removes outlier exposures, stacks residuals to estimate template bias, subtracts expected noise power from residual power, and interpolates and smooths the resulting uncertainty map.
Returns wisp_bias, wisp_err.
interpolate_wisp_nans(wisp, nan_mask, sigma=1)
Fills NaN pixels in one or more wisp maps using Gaussian-smoothed neighboring values and coverage weights.
Utilities
rectify_db(dbname, ndilate=2, mask_jumps=False)
Loads a FITS database file, validates science errors and DQ values, builds a source and quality mask, and returns science, error, DQ, metadata, and mask arrays.
load_stsci_wisp(wisp_path, downsample=4, ext=0)
Loads an STScI wisp reference, downsamples both the template and mask, and keeps the largest connected wisp-mask region.
load_stsci_flat(detector, filter, downsample=4)
Resolves the newest matching CRDS flat for a detector and filter, loads the flat file, and downsamples it.
show_images, plot_comps, and plot_amps
Provide quick-look diagnostics for exposure stacks, NMF components, and fitted amplitudes.
pca(cal, err, mask, wisp_region)
Runs an inverse-variance weighted PCA diagnostic on selected wisp pixels and returns amplitudes and restored 2D components.