Metadata-Version: 2.4
Name: thilos
Version: 0.1.1
Summary: This software is designed to reduce Deep Field Imaging observations obtained with HIPERCAM.
Author-email: "Fabricio M. Pérez-Toledo" <fabricio.perez@gtc.iac.es>
License: GPL-3.0-only
Project-URL: Homepage, https://github.com/Kennicutt/THILOS
Project-URL: Bug Tracker, https://github.com/Kennicutt/THILOS/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: astroalign
Requires-Dist: astrometry-net-client
Requires-Dist: astropy
Requires-Dist: astroquery
Requires-Dist: ccdproc
Requires-Dist: lacosmic
Requires-Dist: loguru
Requires-Dist: matplotlib
Requires-Dist: numpy
Requires-Dist: PyYAML
Requires-Dist: sep
Requires-Dist: pandas
Requires-Dist: opencv-python
Dynamic: license-file

# THILOS

__THILOS__ is a reduction software for the Broad Band Imaging mode of HIPERCAM at GTC.

Developed by __Fabricio M. Pérez-Toledo__

## General Description

**T**he **HI**percam **Lo**ng-exposure **O**bservations **S**olution (THILOS) is a software for the automatic reduction of deep-imaging observations
with long exposure times simultaneously across all five Sloan broad-band filters. The software applies observation-specific reduction steps, ensuring optimized treatment for different data types. Developed with a focus on simplicity and efficiency, **THILOS** streamlines the reduction pipeline, enabling researchers to obtain calibrated data ready for photometric studies.

### Key Reduction Steps:

1. Creation of the __Master Bias__.
2. Creation of the __Master Flat__.
3. Application of master calibration frames to both __standard star__ and __science frames__.
4. Removal of __cosmic rays__.
5. __Sky subtraction__.
6. Alignment of __science frames__.

### Input Requirements:

The software requires the following frames as input:

- __Bias frames__
- __Sky flat frames__
- __Photometric standard star frames__
- __Science frames__

## Outputs

The generated results consist of one image per observed band. For each image, the following corrections and calibrations will have been applied:

- __Bias subtraction__
- __Flat-field correction__ (including fringing correction for the Sloan z band, if applicable)
- __Image alignment and stacking__

To address cosmetic defects, the __LACosmic algorithm__ is used to handle cosmic ray removal.

## Requirements

### Operative System
- __Any__: The software is designed to run within a __Conda environment__, ensuring compatibility across platforms.

### Dependencies
The following Python packages are required (minimum versions specified), however, they will be installed
automatically together the :

    astroalign>=2.6.1
    astrometry_net_client>=0.6.0
    astropy>=7.1.0
    astroquery>=0.4.10
    ccdproc>=2.5.1
    lacosmic>=1.3.0
    loguru>=0.7.3
    matplotlib>=3.10.3
    numpy>=2.3.1
    PyYAML>=6.0.2
    sep>=1.4.1
    pandas>=3.0.3
    opencv-python>=4.13.0.92

### Hardware Requirements
- __RAM__: Minimum 4GB (higher is recommended for large datasets).

## Installation

Installing THILOS is straightforward. Follow these steps:

1. __Activate your Conda environment__ (or create a new one if needed (see below)):
    ```
    conda activate <your_env>

2. __Install THILOS__ using `pip`:
    ```
    pip install thilos

That's it! THILOS is now almost ready to use ;)

### Optional: Creating a New Conda Environment

If you don’t have an existing Conda environment, you can create one specifically for THILOS with the following commands:

    conda create -n thilos_env python=3.11 -y
    conda activate thilos_env
    pip install thilos

## BEFORE YOU USE THILOS (FOR LINUX USERS)

HIPERCAM frames are data cubes, which means that each frame contains full temporal series of images for each of the five Sloan bands. Therefore, before running THILOS, you must split the frames into individual images for each band. This can be done using the joinup script provided by the HIPERCAM software package.

You __MUST__ install HIPERCAM's image for docker. You should follow the HIPERCAM website instructions. https://github.com/HiPERCAM/hipercam

    $ docker build -t hipercam:latest -f Software/hipercam.dockerfile .

The correct way to avoid to log in with hiperuser by default generating a permission incompatibility is using the following command:

    $ docker run -it --rm --user root -v <PathToYourObservationData>:/home/hiperuser/data hipercam:latest

### HOW TO GET INDIVIDUAL FRAMES?

First, you need to extract the frames from compressed file that you have downloaded from the GTC archive, FTP, etc. Then, you place the frames in the `original/` folder of your observation directory. After that, you can run the following command to split the frames into individual images for each band and full temporal series::

    $ joinup data/original/<some.fits>

Now, you need to setup the script (it's IRAF-like command line interface) to split the frames correctly. The next example shows how to setup the script for a specific observation. You can adjust the parameters according to your needs.

    $ first - first frame to process [1]: 
    $ last - last frame to grab [0]: 
    $ trim - do you want to trim edges of windows? [False]: 
    $ ccd - CCD(s) to join up [0 for all] [0]: 
    $ aper - aperture file to generate regions for ds9 ['none' to ignore] [none]: 
    $ bias - bias frame ['none' to ignore] [none]: 
    $ dark - dark frame ['none' to ignore] [none]: 
    $ flat - flat frame ['none' is normal choice with no bias] [none]: 
    $ fmap - fringe map ['none' to ignore] [<PathToFringeMap>/fmap.hcm]: (only if you want to use one) 
    $ msub - subtract median from each window? [False]: 
    $ ndigit - number of digits to use for frame numbers in output names [4]: (Customizable)
    $ dtype - output data type [float32]: 
    $ dmax - maximum allowable amount of data to write out [GB] [100.0]: (Customizable)
    $ nmax - maximum allowable number of frames to write out [10000]: (Customizable)
    $ compress - internal HDU compression to apply [none]: 
    $ odir - directory for the output files [<OutputFolderPath>]: (Customizable)

#### Rename the joinup output file

THILOS requires a specific naming convention for the output files generated by the joinup script. To facilitate this, a script has been added to rename the joinup output files accordingly. This ensures that THILOS can read the files and continue with the reduction process seamlessly. The usage of the script is as follows:

    $ cd path/to/frames
    $ python -m THILOS.script.rename_joinup_fits

## First-Time Setup

Once Conda is set up, you should run __THILOS__ for the first time to create the file `configuration.json` that has to be configured after.

    $ thilos -c

or

    $ thilos --create_config

You must edit the configuration file, which is located in your frame directory.

You need to set the following parameters in the configuration file:

`Optional Setup`: You need to adjust the setup according to your observation. For example, if you are working with Sloan z and need to remove the fringing, you must enable the option in the `Reduction` section and set the `save_fringing` parameter to true. Regarding the alignment can be modified, but they generally work well as they are.

The directory structure must follow the format `<Your_Program>_<Your_OB>/`. Inside this directory, you should have 
a `raw/` folder where the original frames are stored, and a `reduced/` folder where the reduced frames will be saved.

    Your_program/
        configuration.json
        original/
        raw/
        reduced/


## Running THILOS

After saving and updating the configuration file, you can run the command using the following argument. The software will execute successfully.

    $ thilos -e

or

    $ thilos --execute

### Outputs and Results

Once the process is complete, you will find a collection of reduced frames in the `reduced/` folder inside your frame 
directory. The output includes:

A. __Reduced science frames__:
- One version with the sky included.
- One version with the sky subtracted.

B. __Aligned frames__:
- Both sky-included and sky-subtracted versions.

C. __Final reduced science frames__:
- Both sky-included and sky-subtracted versions.

## Project Structure

    THILOS/
        BPM/
            BPM_HIPERCAM.fits -> Unavailable by now
        config/
            configuration.json   -> Configuration file.
        check_files.py           -> It determines which steps can be performed by the pipeline based on the available FITS files.
        aligning_hcam.py   -> Aligns the science frames.
        Color_Codes.py           -> Gives color to the comments
        DRP.py             -> Handles all the sofware and manages the frames.
        reduction_hcam.py  -> Carries out the clean process.

## Note about the frames

The code is designed to work with __HIPERCAM__ frames. They must be in __FITS__ format.

## LICENSE

This software is under __GPL v3.0__ license. More information is available in the
repository.

## CONTACT

- __Email__: [fabricio.perez@gtc.iac.es](fabricio.perez@gtc.iac.es)

- __Repository__: [https://github.com/Kennicutt/THILOS](https://github.com/Kennicutt/THILOS)
