Metadata-Version: 2.4
Name: frankAllSkyCam
Version: 69
Summary: AllSky camera software for Raspberry Pi + Pi HQ Camera: automated capture, watermarking, timelapses, startrails, and day/night cloud & star detection.
Author-email: Francesco Sferlazza <sferlazza@gmail.com>
License: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/sferlix/frankAllSkyCam
Project-URL: Repository, https://github.com/sferlix/frankAllSkyCam
Project-URL: Issues, https://github.com/sferlix/frankAllSkyCam/issues
Keywords: AllSkyCamera,Astronomy,AllSky,Raspberry Pi,libcamera
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: ephem
Requires-Dist: Wand
Requires-Dist: opencv-python-headless
Requires-Dist: Pillow
Requires-Dist: requests
Dynamic: license-file

<p align="center">
  <img src="docs/images/logo.png" alt="frankAllSkyCam logo" width="120">
</p>

# frankAllSkyCam

Open-source AllSky camera software for a **Raspberry Pi + Pi HQ Camera** (or compatible libcamera sensor) with a fisheye lens. Point it at the sky, run it on a cron schedule, and it takes care of the rest:

- Captures a full-sky JPEG every minute or so, with exposure automatically driven by measured or estimated sky brightness (SQM), day or night.
- Watermarks each image with date/time, sun and moon rise/set times, moon phase, visible-planet icons, your own logo/compass, and any extra sensor data you want to show (weather station, temperature, humidity, ...).
- Estimates **cloud cover** and **star count** directly from the image, using different, purpose-built detection for daytime (blue-sky-vs-cloud color analysis) and nighttime (adaptive point-source detection that accounts for the Moon, trees/obstructions, and partial cloud). At night it **calibrates itself to your camera and sky** over the first weeks, with no action needed from you, to tell moonlit cloud from moonlit clear sky.
- Builds nightly **timelapses** (night-only and/or full 24h) and a **startrail** image, and can upload everything to your own website via FTP.
- Optionally drives a **dew heater** (via a network relay or a GPIO-controlled one) based on the gap between internal temperature and dew point, to keep the lens clear.
- A watchdog reboots the Pi automatically if captures ever stall.

Everything runs unattended via cron, installed with a single command.

![Fireball captured by frankAllSkyCam](docs/images/fireball.jpg)

## Quick install

```
pip3 install frankAllSkyCam
```

On newer Raspberry Pi OS (Bookworm and later), `pip` may refuse a system-wide install; if so, use:

```
pip3 install frankAllSkyCam --break-system-packages
```

Then run it once to generate your config:

```
python3 -m frankAllSkyCam
```

This creates `~/frankAllSkyCam/` with a `config.txt` you'll want to edit before going further - see the full walkthrough below.

---

## 1. Prerequisites

Start from a clean, up-to-date Raspberry Pi OS (Lite is fine, no desktop needed):

```
sudo apt update
sudo apt upgrade
```

Make sure `pip` is available:

```
sudo apt install python3-pip
```

Install ImageMagick's development headers (needed to render the moon-phase image):

```
sudo apt install libmagickwand-dev
```

frankAllSkyCam uses the **libcamera** camera stack - as `rpicam-still`/`rpicam-jpeg` on a current Raspberry Pi OS image (Raspberry Pi renamed the `libcamera-apps` package to `rpicam-apps`; a fresh install has `rpicam-*` pre-installed and no `libcamera-*` binaries at all), or as `libcamera-still`/`libcamera-jpeg` on an older release still using the pre-rename package - frankAllSkyCam auto-detects whichever one is actually installed, so either OS release works without any configuration. The older, pre-libcamera `raspistill` is not supported. Check the camera works before going further:

```
rpicam-jpeg -o test.jpg --immediate -n
```

On an older Raspberry Pi OS release, use `libcamera-jpeg` instead.

You should see it capture and leave a `test.jpg` in the current folder.

## 2. Install frankAllSkyCam

```
pip3 install frankAllSkyCam
```

This pulls in all required Python dependencies automatically (`numpy`, `ephem`, `Wand`, `opencv-python-headless`, `Pillow`, `requests`) - nothing else to install by hand. One thing to expect on a fresh Pi: `opencv-python-headless` doesn't always have a prebuilt wheel for every Raspberry Pi OS / Python version combination, and when pip has to fall back to building it from source, that single step can take a long time (tens of minutes) on a Pi. Let it run - it only happens once, not on every `pip install --upgrade`.

Then launch it once, so it can bootstrap your configuration:

```
python3 -m frankAllSkyCam
```

The first run creates:

```
~/frankAllSkyCam/
~/frankAllSkyCam/img/       (captured images, organized by day)
~/frankAllSkyCam/log/       (logs from every scheduled job)
~/frankAllSkyCam/sqm/       (SQM readings, if enabled)
~/frankAllSkyCam/png/       (logo, compass, moon/planet icons - customize freely)
~/frankAllSkyCam/tools/     (optional extras - see "Extra sensors" below)
~/frankAllSkyCam/config.txt
~/frankAllSkyCam/index.html
```

Any of these files can be freely edited - they live outside the installed package, so a future `pip install --upgrade frankAllSkyCam` will never overwrite your customizations. `png/` in particular is where you'd drop your own logo or compass image, matching the filenames already configured in `config.txt`.

## 3. Configure your system

Edit `~/frankAllSkyCam/config.txt` (e.g. `nano ~/frankAllSkyCam/config.txt`). At minimum, set:

```ini
[site]
inte = <name printed on top-center of the image>
latitude = 44.75
longitude = 9.29
elevation = 1150
time_zone = Europe/Rome
```

If you own a **SQM-LE** sky-quality meter, point to it under `[sqm]` (otherwise leave `use_sqm_le = n` and the software estimates SQM from the image itself):

```ini
[sqm]
use_sqm_le = n
ip_address = <ip_address_of_the_SQM_LE>
port = 10001
sqmLog = n
```

### Hosting the image

**Option A - the Pi serves it locally**, via Apache:

```
sudo apt install apache2 -y
sudo mkdir -p /var/www/html/img
sudo mv ~/frankAllSkyCam/index.html /var/www/html/
```

For a full gallery-style site (timelapses, startrails, sky map) rather than just the bare image, grab the `website/` folder from this repository - plain HTML + JS, no PHP required.

**Option B - upload to an external website via FTP.** Configure `[ftp]` in `config.txt`:

```ini
[ftp]
isFTP = True
FTP_server = your_ftpserver.com
FTP_login = your_username
FTP_pass = your_password
FTP_uploadFolder = /your_folder/
FTP_filenameAllSkyImgJPG = /img/allskycam
FTP_fileNameTimelapseMP4 = /video/frankAllSkycam
FTP_fileNameStarTrailJPG = /startrail/startrail.jpg
```

**Second copy of the image (optional).** To upload the same image a second time, to another folder with another file name on the same FTP server, add to `[ftp]` (it also needs `isFTP = True`; create the remote folder first):

```ini
2ndImageFolder = /other_folder/webcam/
2ndImageFile = skycam
```

`2ndImageFolder` is the full remote folder path, `2ndImageFile` the file name (`.jpg` is added if missing). Both are needed: if either is missing or empty, nothing is uploaded a second time.

Leave `isFTP = False` if you don't want any remote upload.

**Sky status JSON (optional).** After every capture frankAllSkyCam writes `~/frankAllSkyCam/sky_status.json` with the numbers of that capture and the sky ephemeris:

```json
{
  "timestamp": "2026-09-19T21:30:00Z",
  "timestamp_local": "2026-09-19T23:30:00+02:00",
  "night_start": "2026-09-19T19:05:14Z",
  "night_end": "2026-09-20T03:28:18Z",
  "sun_rise": "2026-09-20T05:08:41Z",
  "sun_set": "2026-09-20T17:23:02Z",
  "moon_rise": "2026-09-19T13:54:23Z",
  "moon_set": "2026-09-19T22:02:34Z",
  "moon_illumination_pct": 56.8,
  "moon_phase": "Waxing Gibbous",
  "sqm": 20.84,
  "star_count": 86,
  "cloud_cover": 4.5
}
```

Times are UTC. `sqm` and `star_count` are 0 outside a night exposure. To upload it to your website on every capture, add to `[ftp]` (it also needs `isFTP = True`; create the remote folder first):

```ini
isFTPSkyStatus = True
FTP_fileNameSkyStatusJSON = /status/sky_status.json
```

It is off by default.

### Timelapses

```ini
[timelapse]
nightTL = True   # allskycam_night.mp4, sunset to sunrise
fullTL = True    # allskycam_24h.mp4, full day
```

By default the video is smoothed on generation, since captures a minute or more apart otherwise make stars visibly "jump" frame to frame rather than glide:

```ini
smoothMotion = True   # tblend frame-blend for smoother apparent star motion
deflicker = True      # smooths frame-to-frame exposure/brightness variation
```

Both are cheap (no motion estimation, unlike ffmpeg's `minterpolate`, which is too slow on a Pi over hundreds of frames and prone to ghosting on noisy starfields) and safe to leave on. `config.txt`'s `ffmpeg2`/`ffmpeg3` remain a free-form expert escape hatch for extra encoder flags, but the software already applies its own `-vf` for scaling plus these two filters - if you add your own `-vf`/`-filter:v` there, set `smoothMotion`/`deflicker` to `False` first, since ffmpeg errors out on a duplicate `-vf` flag rather than merging them.

`config.txt` is fully commented - text position, font/color, logo/compass/planet icon placement, and max night exposure (`esp_secs`) are all in there and safe to tweak.

## 4. Test it

```
python3 -m frankAllSkyCam
```

If it worked, you'll find the generated JPEG:

1. In a browser, at `http://<your_raspberry_IP>` (if you set up Apache)
2. At `~/frankAllSkyCam/img/<YYYYMMDD>/<file>.jpg`
3. On your remote FTP host, if configured

## 5. Automate it

```
python3 -m frankAllSkyCam.crontab
```

This installs every scheduled job for you: captures (every minute by day, every 2 minutes at night), a watchdog every 15 minutes, nightly startrail and timelapse generation, daily old-image cleanup, and a periodic ephemeris refresh. Re-run it any time (e.g. once a year) to refresh the sunrise/sunset-based capture windows.

**One frame per minute at night** (smoother night timelapses): only the camera time has to fit in the minute, since the analysis, watermark and upload of one frame overlap with the next capture. Set these in `config.txt`, then re-run `python3 -m frankAllSkyCam.crontab`:

```
[exposure]
esp_secs = 50                 # longest night exposure, leaving room in the minute
night_interval_minutes = 1

[sqm]
sqm_interval_minutes = 5      # measure the night sky brightness every 5 minutes, reuse it in between
```

The sky brightness changes slowly at night, so reusing it for a few minutes doesn't affect the exposure, and each measurement it saves frees about 11 seconds of camera time.

A capture that runs long (a slow exposure, or a slow FTP upload) is safe to overlap with the next scheduled one: an internal lock only ever covers the camera-touching part of a run (SQM measurement through the shot itself) - never analysis, watermarking, saving, or upload, none of which touch the camera. If the camera is still genuinely busy when the next run starts, it waits (up to 90s) rather than skipping or colliding with the capture in progress; past that, it assumes the other run is stuck and clears it before proceeding.

Every job's output goes to its own log file under `~/frankAllSkyCam/log/`, so if anything misbehaves, that's the first place to check. Each file holds only the most recent run's output (overwritten every time, not appended) - `capture.log` in particular would otherwise grow forever given how often captures run.

### Enjoy it!

## 6. (Optional) Generate a static obstruction mask

By default, frankAllSkyCam detects foreground obstructions (trees, roof edges, antennas) dynamically on every frame - works out of the box, but can occasionally misjudge on tricky lighting. Once your camera has been running for a few nights, you can instead generate a static, site-specific mask from your own real footage:

```
frankallskycam-generate-mask
```

Requires at least 5 real, **clear-sky** night captures already in `~/frankAllSkyCam/img/<YYYYMMDD>/` - cloudy/hazy frames are automatically screened out (real cloud sitting over part of the sky would otherwise get baked into the mask as if it were a fixed obstruction), so this can take a few nights on a site that doesn't get many fully clear ones. It scans your recent night frames, derives a mask of whatever's consistently obstructed, and saves it to `~/frankAllSkyCam/static_mask.png` (backing up any existing mask first) along with a preview overlay at `~/frankAllSkyCam/static_mask_preview.png` - **review the preview before trusting the mask**, since it highlights in yellow exactly what will be excluded from every future analysis. Once a mask is present, frankAllSkyCam uses it automatically instead of the dynamic per-frame detection; delete `static_mask.png` (or restore the `.bak` this tool saves) to fall back to dynamic detection again.

Re-run the command any time your site's fixed obstructions change (a tree grows, something new gets mounted near the camera). Entirely optional - skip it if the default dynamic detection already works well for your site.

## 7. (Optional) Correct hot pixels and dark current

Every real sensor has a handful of individual defective photosites ("hot pixels") that show up as fixed bright dots in every long night exposure, plus a small amount of diffuse dark current. Two independent, optional corrections are available - both are opt-in (no config needed) and run automatically, before star/cloud detection and the watermark, on every night capture once set up:

**Dark frame subtraction** (the more thorough option) - captures real dark frames with the lens covered and subtracts them from every night frame, cancelling the whole per-pixel dark signal rather than just known coordinates:

```
python -m frankAllSkyCam.capturedarks
```

Run this **with the lens/dome physically covered** - this camera has no shutter, so it can't be automated mid-sequence; the tool prompts you and waits. It captures 5 exposures (5, 15, 30, 45, 60 seconds) x 5 frames each, averaged into one master dark per exposure (averaging reduces the dark frames' own noise without affecting the fixed pattern they're meant to cancel), and saves the result to `~/frankAllSkyCam/darks/`. It also pauses your regular capture schedule and the watchdog for the duration (both are automatically restored afterward, even if the tool is interrupted) so nothing races it for the camera or reboots the Pi mid-session. Re-run it whenever `additional_night_params`, `night_mode` or `night_sharpness` change in `config.txt` - a stale library (captured under different settings) is automatically detected and ignored rather than silently misapplied.

**Hot-pixel coordinate correction** (cheaper fallback, used automatically only if no dark library is present) - corrects a fixed list of known defective pixel coordinates without needing to cover the lens:

```
python -m frankAllSkyCam.calibratehotpixels <folder-of-your-own-night-jpgs> --out hotpixels.json
```
then move `hotpixels.json` into `~/frankAllSkyCam/`. It finds pixel positions that recur at the exact same coordinate across several of your own archived night frames (`~/frankAllSkyCam/img/<date>/`, at least 5) - real stars drift slightly night to night, fixed sensor defects don't. Add `--layout <name>` if you have a preset exclusion mask for your own watermark layout (keeps text/logo edges from being miscounted as recurring defects); omit it and it just won't exclude any region. Coordinates are specific to one physical sensor - never copy `hotpixels.json` between installs.

Both are entirely optional - frankAllSkyCam works fine without either.

## 8. Night cloud detection calibrates itself (nothing to do)

Moonlight makes a clear sky brighter and bluer, and every camera, white balance and site renders a clear night sky slightly differently. To tell a moonlit clear sky from moonlit cloud reliably, frankAllSkyCam learns what a clear night sky looks like **on your own install**, automatically:

- **Collecting:** every night capture (Sun more than 18 degrees below the horizon) adds one small row of measurements - sky brightness per second of exposure, sky color, star count, texture and the Moon's brightness - to `~/frankAllSkyCam/log/night_calibration_samples.csv`. No images are copied and no cron job is added.
- **Calibrating:** once the samples cover a few clear moonless nights and a few clear nights under a bright Moon (typically 1-3 weeks; longer after a cloudy spell, since it needs both), it derives the clear-sky brightness, the effect of the Moon and the clear-sky color for your camera, without anyone labelling frames. The result is saved to `~/frankAllSkyCam/night_calibration.json`, together with the outcome of the last attempt (e.g. `not calibrated: not enough clear moonlit frames yet`).
- **Using it:** with a calibration, night cloud cover compares each frame with the clear sky expected for the current Moon - its brightness and its color (moonlit clear sky is blue, moonlit cloud grey, light-polluted cloud yellow) - and expects fewer stars under a brighter Moon. Until the first calibration exists, the previous night detection is used unchanged.
- **Staying current:** it recalibrates every 30 days from the last 60 days of samples (older samples are deleted, so the file stays at a few MB). Changing the camera settings it depends on (`additional_night_params` such as gain and white balance, `night_sharpness`, or the resolution) automatically discards the old calibration and starts collecting again.

A calibration that fails its plausibility checks is never applied. To start over, delete `night_calibration.json` and `log/night_calibration_samples.csv`.

**Star count on dark nights:** under a dark, moonless sky (once calibrated) dense star fields such as the Milky Way are counted in full instead of being discarded as clutter, so the reported star count is noticeably higher than before on clear moonless nights.

## 9. Brighter, neutral night images

Straight out of the camera, a night frame is dark (the sky background sits around 12% of full brightness) and greenish (the fixed night white balance). After all measurements are taken - exposure feedback, star count, cloud cover and the night calibration all use the frame as captured - frankAllSkyCam brightens the saved image with a fixed curve and balances the sky colour: a slightly cool grey on a dark night, and a natural blue under moonlight, with the Moon itself staying white. The colour correction works in linear light (the camera's tone curve is undone first), so the Milky Way, stars and clouds keep a neutral colour instead of turning pink, and colour noise is smoothed, as is the brightness grain the stretch would otherwise lift along with the sky, without softening faint stars. Night images are brighter and more natural, and clouds lit by light pollution are easier to see. This applies in both exposure modes. It's set in `config.txt`:

```
[night_display]
night_stretch = 0.15        # brightness a typical night sky background is lifted to; 0 keeps frames as captured
night_neutral_sky = True    # balance the night sky colour (cool grey, or blue under the Moon)
```

If the section is missing, these defaults apply. The same curve is used for every frame, so timelapses don't flicker. During twilight the effect fades out, and it leaves a frame untouched when its sky colour can't be balanced (deep blue twilight, where the red channel is already gone in the captured image).

---

## Extra sensors, weather stations, and the dew heater

`~/frankAllSkyCam/tools/` holds small, independent scripts meant for **you to edit** - they live outside the installed package specifically so a `pip install --upgrade` never touches your customizations.

- **`generateExtraData.py`** collects data from your own devices (a weather station, a Shelly smart plug, an I2C temperature/humidity sensor, ...) and writes a single text string that gets watermarked onto the image (enable it via `et_use = y` under `[extra_text]` in `config.txt`). Several ready-made helper functions are included (Ecowitt/WS90-style JSON stations, a Davis Vantage Pro2 plaintext feed, Shelly devices) - uncomment and configure the ones you have in `getData()`.
- A WS90-style weather station is exported by this script as its own JSON (`~/frankAllSkyCam/tools/ws90_data.json`, optionally uploaded with the `[ftp]` section of `generateExtraData.conf`), independent of `sky_status.json`.
- **`generateExtraData.conf`**, sitting next to it, holds your device URLs/IPs and credentials - kept separate from the main `config.txt` on purpose, so this file can be handed to (or edited by) someone who only needs to touch sensor settings.
- The same file can also **switch a dew heater on/off**, based on `(internal temperature - dew point)`, reusing whatever sensor readings you already fetch for the watermark text (no extra network calls). It supports either a network relay (e.g. a Shelly) or a relay wired directly to a Raspberry Pi GPIO pin - see the `[dew_heater]` section in `generateExtraData.conf`.

Both are wired into the crontab automatically by `python3 -m frankAllSkyCam.crontab`.

## For expert users

Exposure duration is predicted from SQM via a small polynomial model, trained from real-world `(SQM, exposure seconds)` pairs stored in:

```
~/frankAllSkyCam/sqmexp.csv
```

Add or adjust pairs to retune the curve for your own site/camera/gain settings - the software interpolates (degree-3 polynomial regression) between the values you provide. The `esp_secs` parameter in `config.txt` always caps the maximum exposure regardless of what the model predicts.

You can also fully customize the `rpicam-still`/`libcamera-still` invocation via `additional_night_params` / `additional_day_params` in `config.txt` - gain, white balance, anything it accepts (just don't set `--shutter`, `--immediate`, `--mode`, `--denoise` or `--sharpness` there - those are fixed by frankAllSkyCam at night, since the ISP's daylight-tuned denoise and sharpening suppress faint stars; `--contrast` has no effect on night captures, which use `--immediate`, and `--mode` pins a true 2x2-binned, full-FOV sensor readout for better low-light sensitivity per pixel - not just a wider `--gain`).

### Alternative exposure strategy: auto_exposure

By default (`exposure_mode = auto_exposure` in `config.txt`'s `[exposure]` section) each run measures the previous capture's own brightness (inside a circular ROI - `[auto_exposure] roi_percent`, excluding fixed dark obstructions near the frame edges) and adjusts the next exposure toward `target_mean`, with no `sqmexp.csv` calibration needed. It reacts a run late to fast sky changes (clouds moving in, moonrise) since the feedback is carried across runs via a small state file, not live. Setting `exposure_mode = sqm_based` switches back to predicting exposure from the pre-calibrated SQM curve above instead. Whichever mode is active, both predictions are logged side by side to `~/frankAllSkyCam/log/exposure_compare.csv` every run, so you can compare them before committing to a switch.

Note this default only applies to a fresh install - if you're upgrading an existing install, your `config.txt` keeps whatever it already has (nothing overwrites it), so add `exposure_mode = auto_exposure` under `[exposure]` yourself if you want to switch.

**Twilight handoff (auto_exposure only):** `min_exposure_secs` stops the feedback loop from collapsing exposure toward zero once it's genuinely dark - but forced blindly right at sunset/sunrise, it used to produce a badly overexposed frame every dusk and dawn, before the loop had a chance to learn the sky was still twilight-bright. Now, while the sun is below the horizon but the loop's own honest prediction is still under `min_exposure_secs`, libcamera's own auto-exposure drives that capture instead (same as daytime), and its real result is harvested to keep the feedback loop warm for when it does take over - the same check applies at both dusk and dawn, sized by the sky's actual measured brightness rather than a fixed clock, so it holds up under real cloud cover too. `[auto_exposure] twilight_guard_deg` is only a cold-start safety strip near the horizon, not what sizes the handoff.

**Saturation guard (auto_exposure only):** if a previous frame came back significantly clipped (a cloud reflecting light pollution at night, say), the plain brightness ratio can't tell how far over target it really was - a clipped mean looks the same whether the true overexposure was 2x or 30x. `[auto_exposure] saturation_clip_frac_threshold` / `saturation_severity_gain` control when and how much harder the next exposure gets cut in that case.

## Requirements

Installed automatically via pip: `numpy`, `ephem`, `Wand`, `opencv-python-headless`, `Pillow`, `requests`. Python 3.9+. See the note under [Install frankAllSkyCam](#2-install-frankallskycam) about `opencv-python-headless` sometimes needing a slow source build on a Pi.

If you use the optional sensor examples in `tools/generateExtraData.py` that read Raspberry Pi hardware directly (CPU temperature, an I2C sensor, GPIO-driven relays), they rely on `gpiozero`, `smbus`/`smbus2`, and `RPi.GPIO` - all pre-installed on Raspberry Pi OS, no extra steps needed.

## Uninstalling

```
pip3 uninstall frankAllSkyCam --break-system-packages
```

This removes only the installed package. It does **not** remove:

- `~/frankAllSkyCam/` - your `config.txt`, captured images, logs, and SQM data. Kept outside the package on purpose, so a `pip install --upgrade` never overwrites your customizations - but that also means an uninstall never touches it.
- The cron jobs `python3 -m frankAllSkyCam.crontab` installed.

If you want a clean removal (e.g. before reinstalling from scratch), remove both yourself:

```
crontab -l | grep -v '#frankAllSkyCam-managed' | crontab -
rm -rf ~/frankAllSkyCam
```

`rm -rf` is irreversible - back up anything you still need from `~/frankAllSkyCam` first (a customized `config.txt`, images you haven't uploaded/FTP'd elsewhere, ...) before running it.

## License

GPLv3 - see [LICENSE](LICENSE).
