{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4164e1ad043be7cd",
   "metadata": {},
   "source": "# Example: processing the test data one command line step at a time\n\nThis walks you through processing a very small collection of images all the way to lightcurves, running each stage of the pipeline yourself as a separate command.\n\n**This is not the usual way to use AutoWISP.** Most users work through the browser interface, `wisp-bui`, which is where you set the configuration and press start; behind it the pipeline works out which stages still need running for which images, does them in the right order, and keeps track of what has already been done. See the [installation instructions](installation.rst) for how to start it. The browser interface carries its own walkthrough of the same test data used here, one button click at a time; it offers it the first time you open the interface, and the **Tutorial** button on the project list starts it again at any point.\n\nDriving the `wisp-*` commands by hand, as below, was  the only way to use AutoWISP in the beginning. It is still worth doing once, because it shows exactly what each stage takes in and what it produces, and it remains the way to re-run a single stage on its own or to build a processing sequence of your own in a script.\n\nWe use a jupyter notebook format for this description to allow this to be closer to a living document that updates along with AutoWISP. All commands starting with `!` or `%` are shell commands, meaning you can execute them in your operating system's terminal (command prompt in Windows) after dropping the leading `!` or `%`."
  },
  {
   "cell_type": "markdown",
   "id": "4a85e857-268a-47b3-9930-5ef355df52e0",
   "metadata": {},
   "source": [
    "### Create a temporary directory to work in (remember to delete it at the end) and enter it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f413ba31-5808-4b3b-8bd3-d12a7723a946",
   "metadata": {},
   "outputs": [],
   "source": [
    "%mkdir autowisp_test\n",
    "%cd autowisp_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75772c71-1532-46f0-92cf-b806d7620ca8",
   "metadata": {},
   "source": "### Download the test data\n\nThe raw images processed in this example, along with the files that processing should produce, are published on Zenodo:\n\n  * https://doi.org/10.5281/zenodo.15727355 always takes you to the newest version;\n  * the version this page was written against is [record 21539385](https://zenodo.org/records/21539385/files/test_data.zip?download=1).\n\nThe download is around 220 MB. Unpacking it gives you the images to process (`RAW`), the configuration used throughout this example (`test.cfg`), the reference maps needed to work out where the images point (`anet_indices`), and several directories holding the results a successful run should produce: `CAL`, `DR`, `LC`, `MASTERS` and `Gaia`. There are a few other odds and ends (`catalog_tests`, `logs`, `autowisp.db`, `master_config.json`, `survey_instruments.json`) which this example does not use.\n\nMove `RAW`, `test.cfg` and `anet_indices` into the directory we just created. Leave the rest where they are: those are what you can compare your own results against."
  },
  {
   "cell_type": "markdown",
   "id": "4233a5e0",
   "source": "### How the images are organized, and what processing produces\n\n`RAW` is split into four subdirectories, one per kind of image, and this is how AutoWISP expects your own data to be arranged as well. Sorting your images this way before you start is worth the trouble, and a good deal easier than untangling them afterwards.\n\n  * **`zero`** -- bias frames: exposures of as near zero length as the camera allows, measuring the level every pixel starts from even when no light has fallen on it.\n  * **`dark`** -- dark frames: exposures as long as the science ones but with no light reaching the detector, measuring the charge that collects in each pixel by itself.\n  * **`flat`** -- flat frames: images of something as uniformly bright as can be managed, measuring how sensitive the system is to light arriving from each direction.\n  * **`object`** -- the images of the night sky you actually want photometry from. If these are not all alike -- different exposure times, a change of filter, anything in the optical path that was altered part way through -- it is usually better to split them further still and process each group on its own.\n\nProcessing then fills in three kinds of file, and the commands below produce them in this order:\n\n  * **Calibrated images**, under `CAL`, in FITS format, each holding three things: the corrected pixel values, an estimate of the uncertainty on each value, and a mask flagging pixels not to be trusted -- saturated ones, ones that received charge spilling over from a saturated neighbor, and so on.\n  * **Data reduction files**, under `DR`, one per calibrated image, in [HDF5](https://www.hdfgroup.org/solutions/hdf5/) format. Each accumulates everything measured about the stars in that one image: their positions, their shapes, their flux measurements, the transformation between sky and image coordinates.\n  * **Lightcurves**, under `LC`, one per star, also HDF5, each holding the time series of every measurement of that star across all the images.\n\nYou will also end up with a `MASTERS` directory holding the stacked master calibration frames and various summary files. Finally, a `logs` directory. Where those go, and what the files in them are called is defined in `test.cfg`.",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "id": "842df7cb",
   "source": "### Adjust `test.cfg` before you start\n\n`test.cfg` is a configuration file: a plain text file of settings, which saves giving them on every command below. Every command takes it with `--config-file` (or `-c` for short).\n\nTwo things in it were written for the machine the test data was produced on and have to be changed. Open it in a text editor and find the `[astrometry]` section. Replace the `anet-indices` line with the full path of the `anet_indices` directory you moved across, written twice:\n\n```\nanet-indices = [/full/path/to/anet_indices, /full/path/to/anet_indices]\n```\n\nAutoWISP expects two sets of reference maps, one for images covering a small patch of sky and one for a large patch; the bundled set does for both here. **Take care to get these paths right: if either does not exist, AutoWISP quietly solves over the internet instead**. This is much slower and needs an astrometry.net account of your own.\n\nThe same file also carries a `gaia-user`, a `gaia-password` and an `anet-api-key` belonging to the authors. Replace them with your own or simply delete those three lines: the Gaia archive can be queried without an account, and no astrometry.net key is needed once the reference maps above are found.\n\nThe rest of `test.cfg` can be left alone. It is worth a look regardless, as it is a compact summary of how a run is configured; [the options documentation](wisp_options.rst) explains what each setting does.",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "id": "1ea4c33289bc866d",
   "metadata": {},
   "source": "### Perform the calibration for the zero frames\n\nThree things stand between the raw pixel values and a measurement of how much light actually fell on the detector: the fixed offset the camera adds to every pixel (the bias), the charge that builds up in the detector even in complete darkness (the dark current), and the fact that different pixels turn light into signal with different efficiency (the flat field). Each is measured from its own kind of calibration image, which is what the six commands below do, before the seventh applies all three to the images of the sky.\n\nNone of that is compulsory. In many applications -- possibly most -- collecting calibration data is impractical and one does without. If that is your situation, skip ahead to calibrating the object frames and give no masters on the command line. The photometry will not be as good as it could have been, but it will work perfectly well.\n\nWhat is not optional is the calibration step itself, even with nothing to correct for. It converts the pixel values from integers to floating point, adds the information the pipeline needs to the header, separates the colors of a color camera, and so on.\n\nHere we have the calibration frames, so we will do the full job. We begin with the bias, measured from \"zero\" frames -- exposures of as near zero length as the camera allows. These are themselves calibrated first, with no masters to apply.\n\n**Settings that matter most** (they apply to all four `wisp-calibrate` commands below): [exposure-start-utc](wisp_options.rst#cmdoption-arg-exposure-start-utc) or [exposure-start-jd](wisp_options.rst#cmdoption-arg-exposure-start-jd), [exposure-seconds](wisp_options.rst#cmdoption-arg-exposure-seconds), [saturation-threshold](wisp_options.rst#cmdoption-arg-saturation-threshold) and [fnum](wisp_options.rst#cmdoption-arg-fnum). Give [gain](wisp_options.rst#cmdoption-arg-gain) as well if you know it -- the uncertainty estimates depend on it. For a color camera you will need `split-channels` or [raw-hdu](wisp_options.rst#cmdoption-arg-raw-hdu), and if your images contain regions that are not sky, [image-area](wisp_options.rst#cmdoption-arg-image-area) says which part is.\n\nThose non-sky regions can often be put to use rather than simply trimmed away. Many detectors read out extra pixels past the physical edge of the sensor, or keep an area shielded from light, and either measures the bias or dark level *as it is at that moment* rather than as it was when the masters were taken -- which matters, because both drift with temperature. AutoWISP can fit a smooth function to such a region and subtract it from each raw frame before the masters are applied, which you ask for with [overscans](wisp_options.rst#cmdoption-arg-overscans). The masters are then left to supply only the pixel-to-pixel differences, which is the part a strip of edge pixels cannot capture. Whether this helps, and which combination works best, depends on the detector."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7fd8230c34bb11c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-calibrate --config-file test.cfg RAW/zero"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc316ba5-07ab-4121-b2d5-ece1db4dcf07",
   "metadata": {},
   "source": [
    "This should produce a directory called `CAL` under which there will be another directory called `zero` which contains the calibrated bias frames."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7fdf2169c1bdd23",
   "metadata": {},
   "source": "### Create a master bias by stacking the calibrated zero frames\n\nAny single bias frame carries the camera's read noise along with the offset we are after. Combining all of them into one image averages that noise down and leaves a clean measure of the level each pixel starts from. That combined image is the master bias, and it is what gets subtracted from everything calibrated from here on.\n\n**Settings that matter most:** [stacked-master-fname](wisp_options.rst#cmdoption-arg-stacked-master-fname), which decides where the master is written and what it is called. It is a pattern rather than a plain name -- `test.cfg` builds it out of the image type and color channel, which is how the same command knows to produce `zero_R.fits` here and `dark_R.fits` two steps later."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdbbb62c835d4b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-stack-to-master -c test.cfg CAL/zero/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa239fc2-1d89-418d-b23b-7e5f99b8b025",
   "metadata": {},
   "source": [
    "This should produce a directory called MASTERS containing a single FITS file, which is the master bias."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cd55d75d9c1f7da",
   "metadata": {},
   "source": "### Perform the calibration for the dark frames\n\nDark frames are exposures of the same length as the science images taken with no light reaching the detector, so what they record is the charge each pixel accumulates on its own over that length of time.\n\nNote that when calibrating the dark frames we wish to subtract the master bias generated above. In order to do this, we specify that the master that was generated above should be used as a master bias by adding `--master-bias \"R:MASTERS/zero_R.fits.fz\"` to our command line"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3b0444858c2afa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-calibrate --config-file test.cfg RAW/dark/*.fits.fz --master-bias \"R:MASTERS/zero_R.fits.fz\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e906fe5e-b9f9-488a-bd89-69fb14c2eb2c",
   "metadata": {},
   "source": [
    "There should now be another directory under `CAL`, called `dark` which contains the calibrated dark frames."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92e45855e30eb179",
   "metadata": {},
   "source": "### Create a master dark by stacking the calibrated dark images\n\nExactly as for the bias, and for the same reason: combining the individual dark frames averages away their noise and leaves a per-pixel measure of how much dark charge collects during an exposure of this length."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "863ce819c517ebc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-stack-to-master -c test.cfg CAL/dark/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f08cd576b6ae37b4",
   "metadata": {},
   "source": "### Perform calibration for flat frames\n\nFlat frames are images of something as uniformly bright as one can manage -- the twilight sky, or an evenly lit screen. Since the scene itself is featureless, any structure that shows up in them belongs to the instrument: vignetting towards the corners, shadows of dust on the optics, and the differing sensitivity of individual pixels.\n\nThose should be corrected both with the master bias and the master dark generated in the previous two steps:"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c476d24d7812ea0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-calibrate --config-file test.cfg RAW/flat/*.fits.fz --master-bias \"R:MASTERS/zero_R.fits.fz\" --master-dark \"R:MASTERS/dark_R.fits.fz\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c2efbc1083f6f9f",
   "metadata": {},
   "source": "### Build a master flat out of the individual calibrated flat frames\n\nNote that creating a master flat uses a different command than creating master bias and dark! The reason is that AutoWISP is built to cope with flats taken on the twilight sky, which is not perfectly uniform, fades fast enough that successive frames differ in overall level, and may have clouds drifting across it. Combining those calls for more care than the straight stack that serves for bias and dark.\n\nPart of that care is separating the well-exposed frames from the faint ones, which is why there are two outputs rather than one: [high-flat-master-fname](wisp_options.rst#cmdoption-arg-high-flat-master-fname) and [low-flat-master-fname](wisp_options.rst#cmdoption-arg-low-flat-master-fname). The high-illumination master is only produced at all if at least [min-high-combine](wisp_options.rst#cmdoption-arg-min-high-combine) frames survive the checks."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36d2de90a9395c65",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-stack-to-master-flat -c test.cfg CAL/flat/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75c74545b22c9a35",
   "metadata": {},
   "source": "### Perform calibration for object frames\n\nThese are the actual images of the night sky, the ones we want photometry from. Now we use all three masters: the bias and the dark are subtracted and what remains is divided by the flat. The result is what every step from here on measures."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44cf34070c00aa9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-calibrate --config-file test.cfg RAW/object/*.fits.fz --master-bias \"R:MASTERS/zero_R.fits.fz\" --master-dark \"R:MASTERS/dark_R.fits.fz\" --master-flat \"R:MASTERS/flat_R.fits.fz\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8163a050-f500-4138-a8da-563a416f4be8",
   "metadata": {},
   "source": [
    "If you wish to see how your calibrated images look, we recommend using [SAO DS9](https://sites.google.com/cfa.harvard.edu/saoimageds9/download)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "954c24e99b3697c3",
   "metadata": {},
   "source": "### Find stars in the calibrated object frames\n\nBefore any brightness can be measured, AutoWISP needs to know whereabouts on each image the stars actually fell. This step goes through each calibrated image looking for anything standing out above the local background and records, for each thing it finds, an approximate position, a rough flux, and an elliptical Gaussian fitted to the pixels around it as a crude description of its shape. The positions are not yet precise -- making them so is the job of the next step -- but they are good enough to start from. Behind the scenes the search is done by `fistar`, from the [FITSH](https://ui.adsabs.harvard.edu/abs/2012MNRAS.421.1825P/abstract) package.\n\nThis will create another directory called `DR` which contains what we call data reduction files. These files are where the pipeline saves the results of processing the calibrated images, and every stage from here on adds to them. So far, they will only contain the stars that we find in the images. If you wish to see the contents of HDF5 files, the best tool we are aware of is: [HDFView](https://www.hdfgroup.org/download-hdfview/)\n\n**Settings that matter most:** [brightness-threshold](wisp_options.rst#cmdoption-arg-brightness-threshold), which decides how far above the background something has to stand to count as a source, and, if needed, [filter-sources](wisp_options.rst#cmdoption-arg-filter-sources) and [srcextract-max-sources](wisp_options.rst#cmdoption-arg-srcextract-max-sources) to throw out what you do not want and cap how much is kept.\n\nFinding a good threshold is a matter of trial and error, and the browser interface has a sandbox for exactly that: it runs source extraction on any calibrated image with whatever values of those three settings you care to try, and shows the result straight away."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4380087219f6dc02",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-find-stars --config-file test.cfg CAL/object/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b61f312c826feb6",
   "metadata": {},
   "source": "### Perform astrometry on DR files\nThis matches the extracted sources to a catalog of stars from the Gaia space mission and uses that information to figure out how to convert from coordinates on the sky to coordinates within each image. The conversion will also be stored in the DR files. Having it is what lets every later step use the catalog's far more precise positions, and everything else the catalog knows about each star, instead of relying on what can be measured from the images alone.\n\nIt happens in two stages. First astrometry.net matches the few tens to few hundred brightest extracted sources against the sky, which gives a rough transformation to start from. AutoWISP then refines that repeatedly until very nearly every extracted source has been paired with its catalog counterpart -- thousands of them in a wide-field image. Having that many pairs to fit is what makes it possible to model the substantial distortion wide-field optics usually have, rather than just a shift, a rotation and a scale.\n\nThis is the step that uses the `anet_indices` you configured above. It also asks the Gaia archive, over the internet, which stars ought to be in each image, so a working connection is needed. Each patch of sky is fetched only once and kept under `MASTERS/Gaia`, so running this again is much quicker.\n\nDo not be alarmed if a frame fails: in this dataset one of the ten object images cannot be solved. That is a normal outcome rather than a fault -- AutoWISP records the failure, leaves that image out of everything that follows, and carries on with the other nine.\n\n**Settings that matter most:** [anet-indices](wisp_options.rst#cmdoption-arg-anet-indices), and [anet-api-key](wisp_options.rst#cmdoption-arg-anet-api-key) if you are solving over the web, both discussed above. Then [astrometry-catalog-max-magnitude](wisp_options.rst#cmdoption-arg-astrometry-catalog-max-magnitude), which should roughly correspond to the brightness threshold used for source extraction, so that the catalog holds about the same stars the images do; [frame-fov-estimate](wisp_options.rst#cmdoption-arg-frame-fov-estimate), telling the solver roughly how much sky an image covers; [astrometry-order](wisp_options.rst#cmdoption-arg-astrometry-order), how much distortion the transformation is allowed to describe; and [max-srcmatch-distance](wisp_options.rst#cmdoption-arg-max-srcmatch-distance), [min-match-fraction](wisp_options.rst#cmdoption-arg-min-match-fraction) and [max-rms-distance](wisp_options.rst#cmdoption-arg-max-rms-distance), which between them decide what counts as a match and when a solution is good enough to accept."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f52b6a6966916ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-solve-astrometry --config-file test.cfg DR/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fca906468321545",
   "metadata": {},
   "source": "### Perform PSF/PRF fitting\n\nA star is a point source, but by the time its light has passed through the atmosphere and the optics and landed on the detector it is spread into a small blob of a characteristic shape. This step models that shape, and at the same time finds the factor by which it has to be scaled to reproduce the pixels each star actually covers. That factor is a measure of the star's brightness, and it is the first of the two kinds of brightness measurement AutoWISP produces.\n\nThe shape is not fitted independently for every star: it is required to vary smoothly across each image and from one image to the next, so that thousands of stars contribute to a model with comparatively few free parameters. The fitting itself is done by [AstroWISP](https://kpenev.github.io/AstroWISP/), the lower-level library AutoWISP is built on, which models the shape as a piecewise bi-cubic function.\n\nThe two names in the heading are not interchangeable:\n\n  * The **point spread function**, PSF(dx, dy), is how much light arrives at an offset (dx, dy) from where the star is projected. To predict what a given pixel will record you have to integrate it, multiplied by a map of how sensitivity varies within a pixel, over the area of that pixel.\n  * The **pixel response function**, PRF(dx, dy), goes one step further and gives directly what a pixel whose center sits at offset (dx, dy) will register -- for any real dx and dy, not just whole pixels. It has the sub-pixel sensitivity map and other detector behaviour already folded into its definition. Being in effect the PSF already convolved with that map, it is the smoother of the two and the easier to model.\n\n**Settings that matter most:** [shape-mode](wisp_options.rst#cmdoption-arg-shape-mode) picks which of the two you are fitting; [shape-grid](wisp_options.rst#cmdoption-arg-shape-grid) and [shape-terms-expression](wisp_options.rst#cmdoption-arg-shape-terms-expression) set how much freedom the shape has, across a star and across the image respectively; [shapefit-smoothing](wisp_options.rst#cmdoption-arg-shapefit-smoothing) and [map-variables](wisp_options.rst#cmdoption-arg-map-variables) control the fit further; [background-annulus](wisp_options.rst#cmdoption-arg-background-annulus) says where the sky level around each star is measured; [photometry-catalog-max-magnitude](wisp_options.rst#cmdoption-arg-photometry-catalog-max-magnitude) sets how faint a star still gets measured, and going too deep costs a great deal of time and disk space for very noisy measurements; and [subpixmap](wisp_options.rst#cmdoption-arg-subpixmap) is needed for a color camera."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68ec099d065705b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-fit-star-shape --config test.cfg CAL/object/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77db999f3eb73eaf",
   "metadata": {},
   "source": "### Perform aperture photometry\n\nThe second and more direct way of measuring brightness: add up the light inside circles of several sizes drawn around each star's position -- four sizes in this example. Pixels lying only partly inside a circle have to be split correctly, which needs both the shape fitted in the previous step and the sub-pixel sensitivity map.\n\nWhich size works best is not known in advance. A small circle collects less of the sky background but also less of the star, and a large one is more likely to catch a neighbor; the best compromise depends on how sharp the images are and how crowded the field is. So all four are measured, kept side by side, and can be compared later.\n\n**Settings that matter most:** [apertures](wisp_options.rst#cmdoption-arg-apertures), the list of radii to measure, and [subpixmap](wisp_options.rst#cmdoption-arg-subpixmap) for a color camera."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d48e0f61777858f0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-measure-aperture-photometry --config test.cfg CAL/object/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "484ffdf7663abc76",
   "metadata": {},
   "source": "### Run fit source extracted PSF map\n\nThis is not a repeat of the PSF/PRF fitting step above, and the two should not be confused. That step built a detailed model of the shape of a star directly from the pixels. This one works with something much cruder that we already have: back in the find-stars step, `fistar` fitted an elliptical Gaussian to the pixels belonging to each star it found, boiling the shape of that star down to three numbers -- here `S`, `D` and `K`, one describing the size and two the elongation. What this step does is smooth those three numbers.\n\nFitted star by star, they are noisy. But the shape really does vary smoothly across an image, so a smooth function of position -- and, if you wish, of catalog properties such as brightness or color -- is fitted through all the stars at once, and a much better estimate for each individual star is then read back off it. The steps that remove instrumental trends later on use these smoothed values rather than the noisy per-star ones.\n\nThe detailed PSF/PRF model is not used for this, for two reasons: it does not currently report summary numbers of this kind, and in many setups it is too time-consuming to run in earnest and is replaced by a dummy fit.\n\n**Settings that matter most:** [srcextract-psf-params](wisp_options.rst#cmdoption-arg-srcextract-psf-params), naming which of the extracted shape parameters to smooth -- `S`, `D` and `K` here -- and [srcextract-psfmap-terms](wisp_options.rst#cmdoption-arg-srcextract-psfmap-terms), which sets what the smooth function is allowed to depend on."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac02ceb938936395",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-fit-source-extracted-psf-map --config test.cfg DR/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6245986297f78f41",
   "metadata": {},
   "source": "### Perform magnitude fitting\n\nImages taken from the ground are not on a common brightness scale. The amount of atmosphere in the way changes as the target rises and sets. Its transparency changes with humidity and cloud cover, and the instrument contributes effects of its own. All of this makes the same star come out brighter in one frame than another for reasons that have nothing to do with the star. This step removes everything of that sort that varies smoothly with the properties of the star -- its position on the frame, its brightness, and its color.\n\nTo do this, AutoWISP asks you to pick one frame that looks particularly good -- sharp images, a transparent sky, a dark background -- and take it as the reference. For every other frame a smooth correction is then derived that brings its measurements as close to the reference as they will go. Once all the frames have been corrected, they are combined into a reference of much higher signal-to-noise than any single frame could be, and the whole procedure is repeated against that. Here it runs six times, which is why `MASTERS` ends up with six `mphotref` files and six sets of statistics.\n\n**Settings that matter most:** [single-photref-dr-fname](wisp_options.rst#cmdoption-arg-single-photref-dr-fname): names the frame to start from; [correction-parametrization](wisp_options.rst#cmdoption-arg-correction-parametrization): defines what the smooth correction is allowed to depend on; and [magfit-catalog-max-magnitude](wisp_options.rst#cmdoption-arg-magfit-catalog-max-magnitude): usually should have the same value as used for the photometry catalog."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97be5036a9282077",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-fit-magnitudes --config test.cfg DR/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e2c2a6-4ff0-4509-a014-2d7ebe74568f",
   "metadata": {},
   "source": "One of the products of this step is a series of magnitude fitting statistics files, which store information about most of the stars that are in the images. These are human readable (white-space separated files) that you can view with a text editor or open in your favorite application for manipulating data organized in columns. The information in these files is a good guide on how good the photometry is. \n\nFor each star, the first half of the columns in the statistics files are as follows:\n* For the first aperture:\n  * The number of brightness measurements of the star\n  * The number of brightness measurements that are not outliers (unusually high or low)\n  * The median of all non-outlier brightness measurements\n  * A first measure of the uncertainty in the brightness measurements, estimated as the root-mean-square deviation from the median.\n  * A second measure of the uncertainty in these measurements, estimated as the median of the absolute deviation around the median.\n* Repeat for the second aperture.\n* ...\n\nThe second half of the columns are meant for internal use, and their definition may change in the future."
  },
  {
   "cell_type": "markdown",
   "id": "b5c7ce473c9c342b",
   "metadata": {},
   "source": "### Create light curves\n\nEverything so far has been organized by image: one file per frame, holding every star in it. To study how a single star behaves over time you need the opposite arrangement, so this step is essentially a transpose. It gathers each star's measurements from all the frames into a file of its own under `LC` -- a lightcurve. Measurements from earlier runs, perhaps of a slightly different pointing or a different instrument setup, can be added to the same file.\n\n**Settings that matter most:** [lc-catalog-max-magnitude](wisp_options.rst#cmdoption-arg-lc-catalog-max-magnitude), which usually wants the same value as the photometry catalog; [latitude-deg](wisp_options.rst#cmdoption-arg-latitude-deg), [longitude-deg](wisp_options.rst#cmdoption-arg-longitude-deg) and [altitude-meters](wisp_options.rst#cmdoption-arg-altitude-meters), which say where the observations were made from and are what the barycentric times are computed with; and [max-memory](wisp_options.rst#cmdoption-arg-max-memory), which caps how much is held in memory before being written out. That last one matters here in a way it does not elsewhere: this step touches every lightcurve and every data reduction file at once, and a survey can have tens of thousands of them."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3eb83c914233192",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-create-lightcurves --config test.cfg DR/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87513585d9b89842",
   "metadata": {},
   "source": "### Perform EPD on light curves\n\nEPD stands for External Parameter Decorrelation. Some of the scatter left in a lightcurve is not random: it tracks quantities that have nothing to do with the star, such as where on the detector it happened to fall, how bright the sky background was, how sharp the images were, or how far from overhead the telescope was pointing. This step works out, for each lightcurve on its own, the linear combination of such quantities that accounts for the most of its variation, and subtracts that off.\n\nChoose those quantities with care. If you include anything that varies on the same timescale as the signal you are hoping to find, it will remove part of the signal along with the noise.\n\n**Settings that matter most:** [variables](wisp_options.rst#cmdoption-arg-variables) gives names to the lightcurve quantities you want to work with, and [epd-terms-expression](wisp_options.rst#cmdoption-arg-epd-terms-expression) then builds the terms to fit out of those names -- in this example just one, the airmass, written as `O1{1/cos(z)}`. [epd-datasets](wisp_options.rst#cmdoption-arg-epd-datasets) says which measurements are corrected and where the corrected version is written. `test.cfg` leaves it unset, which is usually what you want: every aperture the photometry produced is then corrected, along with the shape-fitted magnitudes if the star shape was really fit rather than assumed constant across each star. Set it yourself only to correct some smaller selection of the measurements."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39f2d44fb813068a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-epd --config test.cfg LC/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "615913353e67f6e2",
   "metadata": {},
   "source": "### Generate EPD statistics file for light curves\n\nThis summarizes how much scatter each lightcurve has left after EPD. Besides being a good way to judge how the processing has gone, the file is what the next step reads in order to pick out the stars that are quiet enough to serve as its templates -- so it has to be produced before TFA can be run."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c83fa0d87e1da5bc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-generate-epd-statistics --config test.cfg LC/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c819b2640ac2e6cb",
   "metadata": {},
   "source": "### Perform TFA on light curves\n\nThe Trend Filtering Algorithm takes out of each star's lightcurve whatever it has in common with a set of other stars' lightcurves. The reasoning is that a real variation belongs to one star alone, whereas anything the instrument or the atmosphere does tends to show up in many stars at once.\n\nThe same warning as for EPD applies, with more force: TFA can distort a genuine signal or remove it altogether. Where the shape of the signal being looked for is known in advance, there are variants of the procedure that preserve it.\n\n**Settings that matter most:** [tfa-sqrt-num-templates](wisp_options.rst#cmdoption-arg-tfa-sqrt-num-templates) sets how many stars are used as templates -- its square, so 2 here means 4 -- and [tfa-faint-mag-limit](wisp_options.rst#cmdoption-arg-tfa-faint-mag-limit) keeps stars too faint to be useful out of that selection. [tfa-datasets](wisp_options.rst#cmdoption-arg-tfa-datasets) is the counterpart of `epd-datasets`, and by default takes the EPD-corrected measurements as its input."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a07df4bbd6eb1ff5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wisp-tfa --config test.cfg LC/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "484891cbf62aa462",
   "metadata": {},
   "source": "### Generate TFA statistics file for light curves\n\nThe same summary as before, this time computed after TFA. Comparing the two files shows how much TFA actually gained, star by star."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cded61e3310af4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-generate-tfa-statistics --config test.cfg LC/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "552ca160-8ceb-4f26-81f4-231a8a6ea114",
   "metadata": {},
   "source": "### Extract brightness vs time from lightcurves\n\nThe lightcurves generated above are in HDF5 format. You can open them directly using HDFView as before. We have attempted to organize the content in a logical way. However, these files may be somewhat cumbersome to work with, and likely contain a lot more entries than a typical user would care about. As an example, let us collect all aperture photometry measurements from one of the newly generated lightcurves in a CSV file.\n\nRunning `wisp-get-from-lc --list-lc-quantities` prints everything that can be asked for this way, with a short description of each."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09b230bb-3a47-4cff-b651-63c549e933d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wisp-get-from-lc LC/GDR3_809281106831668736.h5 --expression bjd=skypos.BJD --expression 'mag{aperture_index}=apphot.magfit.magnitude' -o GDR3_809281106831668736.csv --find-best"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef848ea-c6ee-493b-aa67-7b75c76fe3b2",
   "metadata": {},
   "source": [
    "You can open this file with your favorite text editor or load it in your favorite software for manipulating column-organized data. The contents of the file should look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "434a4d1f-3f22-4ecd-9237-d2182280082b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bjd,mag0,mag1,mag2,mag3\n",
      "2457819.837050383,4.006945615264255,3.17406685414204,3.0312522103547104,2.981423791605317\n",
      "2457819.837690473,3.743435615264255,3.0646768541420397,2.9453222103547105,2.886863791605317\n",
      "2457819.838957673,4.121655615264255,3.17709685414204,3.0010422103547105,2.962123791605317\n",
      "2457819.839599043,4.031455615264255,3.18722685414204,3.0456322103547104,3.044273791605317\n",
      "2457819.840274193,3.945705615264255,3.15156685414204,2.9139522103547106,2.827073791605317\n",
      "2457819.840906103,4.124025615264255,3.26076685414204,3.0048822103547104,2.940343791605317\n",
      "2457819.841551873,4.051135615264255,3.19773685414204,2.9911622103547106,2.912093791605317\n",
      "2457819.842187163,3.990045615264255,3.12918685414204,2.9628322103547107,2.919363791605317\n",
      "2457819.842825823,3.761285615264255,3.04416685414204,2.8789522103547105,2.860253791605317\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('GDR3_809281106831668736.csv', 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05302956-f53d-4da8-b355-0970b82e110f",
   "metadata": {},
   "source": "There are nine rows in the file, one for each of the science images that made it through processing. The tenth is the frame astrometry could not solve, mentioned earlier.\n\n  * the first column (bjd) is the Barycentric Julian Date. This is a commonly used way of representing time in astronomy in units of days. It compensates for different light travel time from different stars to the Earth as the Earth revolves around the Sun, compensating for the fact that a given astronomical event will be observed to occur earlier if the earth is closer to the source than the sun and later if it is further.\n  * The remaining four columns are the brightness of the star measured through each of the four apertures used in this example, in magnitudes -- the scale astronomers use, on which smaller numbers mean brighter -- as they come out of the magnitude fitting step, which put all the images on a common scale.\n\nWhat we asked for above was `apphot.magfit.magnitude`. To see instead what EPD and TFA made of those measurements, ask for `apphot.epd.magnitude` or `apphot.tfa.magnitude` in the same way."
  },
  {
   "cell_type": "markdown",
   "id": "4d9d5700-c675-4828-893e-f49644bef037",
   "metadata": {},
   "source": [
    "***When you are done reviewing your results, remember to delete the directory you created at the beginning***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:base] *",
   "language": "python",
   "name": "conda-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
