FITS File Format Specification
==============================

This document specifies the expected structure and format of FITS monitoring files that can be read by ``FITSMonitoringSource``.

Overview
--------

FITS monitoring files for CTAO DL0 are organized with:

* **Primary HDU** – global metadata
* **Binary table extensions** – monitoring data for each source/group pair

Each monitoring source can have multiple monitorable groups. FITS headers are generated from :class:`~ctao_datamodel.models.dataproducts.Product` instances.

File Structure
--------------

.. code-block:: text

    Primary HDU

    Extension 1 (BinTableHDU)
        timestamp + data columns for source/group 1

    Extension 2 (BinTableHDU)
        timestamp + data columns for source/group 2

    ...more as needed

Binary Table Extensions
~~~~~~~~~~~~~~~~~~~~~~~

Each extension must have:

* **timestamp column** – stored as elapsed time relative to ``MJDREF`` in ``TIMEUNIT``; automatically reconstructed as :class:`~astropy.time.Time` by column headers
* **Data columns** – matching the container definition with correct units

Timestamp serialization rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Timestamps must always be written as elapsed time from the selected epoch:

* Header values: ``(time - MJDREF).to_value(TIMEUNIT)``
* Column values: ``(time - MJDREF).to(TIMEUNIT)``

Required Header Keywords
------------------------

CTAO Data Model Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All FITS headers are generated from :class:`~ctao_datamodel.models.dataproducts.Product` instances, which ensures compliance with the CTAO data model. Key metadata includes:

.. csv-table::
    :header: "Field", "Type", "Example", "Description"

    "``data.type``", "``DataType`` enum", "``DataType.Pointing``", "Identifies the monitoring source (e.g., Pointing, Weather, FRAM)"
    "``data.level``", "``DataLevel`` enum", "``DataLevel.DL0``", "Data processing level (always DL0 for monitoring)"
    "``data.division``", "``DataDivision`` enum", "``DataDivision.MONITORING``", "Data division (always MONITORING for monitoring data)"
    "``instance.assembly_name``", "String", "``drive``", "**Optional.** Name of the monitorable group. Used to disambiguate when multiple groups from the same source are present"
    "``observation.coverage.time``", "``TimeCoverage``", "``t_min=1767290400.0, t_max=1767333600.0``", "Time coverage stored as elapsed values in ``TIMEUNIT`` relative to ``MJDREF`` (example: 2026-01-01 18:00 to 2026-01-02 06:00 UTC with default ``MJDREF=40587`` and ``TIMEUNIT=s``)"
    "``curation``, ``contact``, ``model``", "Metadata", "(see ctao_datamodel)", "Standard CTAO metadata (license, author, data model version, etc.)"

The source name used for registry lookup is constructed as::

    source_name = f"{product.data.type}{product.data.level}{product.data.division}"

For example, a ``Product`` with ``DataType.Pointing``, ``DataLevel.DL0``, and ``DataDivision.MONITORING`` creates the source name ``"PointingDL0Monitoring"``, which is matched against registered :class:`~ctapipe_mon_acada.fitsdataformat.MonitoringSourceInfo` subclasses.

Writing FITS Files with write_monitoring_fits
----------------------------------------------

The :func:`~ctapipe_mon_acada.testing.write_monitoring_fits` function provides a high-level API for creating spec-compliant FITS monitoring files. It handles:

* FITS header generation from :class:`~ctao_datamodel.models.dataproducts.Product` instances
* Automatic conversion of :class:`~astropy.time.Time` columns to elapsed values relative to ``MJDREF`` in ``TIMEUNIT``

* Per-column FITS time headers (``TCTYP``, ``TCUNI``, ``TUNIT``) for automatic Time reconstruction
* Compliance with CTAO data model standards

Example: Writing a Single Monitoring Group
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    from astropy.table import Table
    from astropy.time import Time
    import astropy.units as u
    import numpy as np
    import ctao_datamodel.models.dataproducts as dp
    from ctapipe_mon_acada.testing import make_monitoring_product, write_monitoring_fits

    # Create sample data
    t0 = Time("2020-01-01T00:00:00", scale="utc")
    offsets = np.arange(3) * u.s

    pointing_table = Table({
        "timestamp": (t0 + offsets),
        "azimuth": np.array([45.0, 45.1, 45.2]) * u.deg,
        "altitude": np.array([60.0, 60.1, 60.2]) * u.deg,
    })

    # Create a valid Product instance
    product = make_monitoring_product(
        datatype=dp.DataType.POINTING,
        assembly_name="drive",
        tstart=t0,
        tstop=(t0 + 2 * u.s),
    )

    write_monitoring_fits("data.fits", pointing_table, product)

Example: Multi-Group File
~~~~~~~~~~~~~~~~~~~~~~~~~~

For files with multiple groups from the same source, write each group separately and combine them:

.. code-block:: python

    from astropy.io import fits
    from ctapipe_mon_acada.testing import (
        _apply_timestamp_column_headers,
        _timestamp_to_elapsed,
        _product_to_header,
    )

    hdus = [fits.PrimaryHDU()]
    for table, hdu_name, product in groups:
        fits_table = table.copy()
        fits_table["timestamp"] = _timestamp_to_elapsed(fits_table["timestamp"], product)
        hdu = fits.BinTableHDU(data=fits_table, name=hdu_name)
        col_names = [c.name for c in hdu.columns]
        if "timestamp" in col_names:
            _apply_timestamp_column_headers(
                hdu.header, col_names.index("timestamp") + 1
            )
        hdu.header.update(_product_to_header(product))
        hdus.append(hdu)

    fits.HDUList(hdus).writeto("multigroup.fits", overwrite=True)

Testing Utilities
-----------------

The :mod:`ctapipe_mon_acada.testing` module provides utilities for creating valid test fixtures without manually constructing :class:`~ctao_datamodel.models.dataproducts.Product` instances.

make_monitoring_product()
~~~~~~~~~~~~~~~~~~~~~~~~~

:func:`~ctapipe_mon_acada.testing.make_monitoring_product` creates minimal, fully-valid ``Product`` instances for test fixtures. It:

* Automatically sets required metadata (license, model, contact information)
* Constructs the source name from ``DataType``, ``DataLevel``, and ``DataDivision``
* Embeds time coverage as elapsed time relative to ``MJDREF`` in ``TIMEUNIT``. The time coverage is made optional in order to allow invalid data generation for testing purposes, but it is mandatory to include it for realistic test fixtures.

Why ``t_min``/``t_max`` and not ``tstart``/``tstop``?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``tstart`` and ``tstop`` are the helper function inputs and are passed as
absolute :class:`~astropy.time.Time` objects.

The CTAO data model stores the corresponding values in
``observation.coverage.time`` as ``t_min`` and ``t_max`` (elapsed values),
together with the time reference (``MJDREF``/``TIMESYS``/``TIMEUNIT``).

In other words, the helper performs the mapping:

* input: ``tstart`` / ``tstop`` (absolute Time)
* stored: ``t_min`` / ``t_max`` where
    ``t_min = (tstart - MJDREF).to_value(TIMEUNIT)`` and
    ``t_max = (tstop - MJDREF).to_value(TIMEUNIT)``

.. code-block:: python

    from ctao_datamodel.models.dataproducts import DataType
    from ctapipe_mon_acada.testing import make_monitoring_product

    # Minimal product (no time coverage)
    product = make_monitoring_product(
        datatype=DataType.Pointing,
        assembly_name="drive",
    )

    # Product with time coverage from absolute Time objects
    from astropy.time import Time

    product = make_monitoring_product(
        datatype=DataType.Weather,
        assembly_name="weather_station_01",
        tstart=Time("2021-01-01T00:00:00", scale="utc"),
        tstop=Time("2021-01-02T00:00:00", scale="utc"),
    )

Integration with write_monitoring_fits()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:func:`~ctapipe_mon_acada.testing.make_monitoring_product` is designed to pair seamlessly with :func:`~ctapipe_mon_acada.testing.write_monitoring_fits`:

.. code-block:: python

    from astropy.table import Table
    from astropy.time import Time
    import astropy.units as u
    import numpy as np
    from ctapipe_mon_acada.testing import write_monitoring_fits
    from ctapipe_mon_acada.testing import make_monitoring_product
    import ctao_datamodel.models.dataproducts as dp

    # Create test data
    t0 = Time("2021-01-01T00:00:00", scale="utc")
    offsets = np.arange(3) * u.s
    table = Table({
        "timestamp": (t0 + offsets),
        "temperature": np.array([5.0, 5.1, 5.2]) * u.deg_C,
        "pressure": np.array([780.0, 780.1, 780.2]) * u.hPa,
        "humidity": np.array([0.25, 0.26, 0.27]),
    })

    # Create valid product
    product = make_monitoring_product(
        datatype=dp.DataType.WEATHER,
        assembly_name="default",
        tstart=Time("2021-01-01T00:00:00", scale="utc"),
        tstop=Time("2021-01-02T00:00:00", scale="utc"),
    )

    # Write to FITS
    write_monitoring_fits("test_weather.fits", table, product)

Default Metadata Constants
^^^^^^^^^^^^^^^^^^^^^^^^^^^

The testing module pre-defines common metadata values:

.. code-block:: python

    DEFAULT_CURATION = dp.Curation(
        license="CC-BY-4.0",
        license_url="https://creativecommons.org/licenses/by/4.0/",
    )
    DEFAULT_MODEL = dp.DataModel(
        name="CTAO-DL0",
        version="1.0.0",
        url="https://ctao.org",
    )
    DEFAULT_CONTACT = dp.Contact(
        name="Test Author",
        organization="Test Org",
        email="test@ctao.org",
    )
