Metadata-Version: 2.4
Name: dofjson
Version: 0.1.0
Summary: Prototype client for SIDOF's undocumented JSON open data service for Mexico's official gazette (DOF).
License: Apache-2.0
Project-URL: Homepage, https://github.com/INGEOTEC/LegalIA/tree/main/packages/dofjson
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31
Requires-Dist: pypdf>=3.0
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"

# dofjson

Prototype client for the JSON open data service exposed by
[sidof.segob.gob.mx](https://sidof.segob.gob.mx/datos_abiertos), the
Secretaría de Gobernación's system for Mexico's official gazette (DOF,
Diario Oficial de la Federación).

The service's public docs only show sample responses, but its real,
unauthenticated endpoints were found under `https://sidof.segob.gob.mx/dof/sidof/`:

| Endpoint | Description |
|---|---|
| `GET /diarios/porFecha/DD-MM-YYYY` | Edition metadata for a date (Matutina/Vespertina/Extraordinaria) |
| `GET /notas/DD-MM-YYYY` | Notes/documents published on a date |
| `GET /notas/nota/{codNota}` | Full detail of a single note, including its HTML content |
| `GET /indicadores/DD-MM-YYYY` | Economic indicators (exchange rate, TIIE, UDIS) |

This is an experimental package for evaluating whether this service is a
viable alternative (or complement) to `dof2md`'s PDF download + Markdown
conversion pipeline — notes already come with structured HTML content,
which may be easier to work with than OCR'd PDFs.

On top of the raw endpoints, the client offers note-scoped downloads that
resolve a note's page span (`infer_paginas`) and fetch it in whichever form
you want:

- `download_nota_imagenes(codNota)` — the note's scanned page image(s).
- `download_nota_pdf(codNota)` — the note as its own PDF: the whole edition
  PDF (there is no per-note PDF endpoint) sliced to just the note's pages,
  using `pypdf`.

## Usage

```bash
pip install -e "packages/dofjson[test]"
dofjson 2026-07-16 --endpoint notas --outdir output
```

## Building a local archive of daily indexes (`--archivo`)

`dofjson --archivo` downloads the **daily notes index** incrementally, day by
day, over a whole date range (by default from January 2, 1917 to today). For
each date it does exactly what `dofjson YYYY-MM-DD --endpoint notas` does —
`get_notas(date)` filtered with `quita_notas_sin_titulo` — and saves one JSON
per day. It does **not** download each note's content or scanned images: only
the index.

```bash
dofjson --archivo                                      # 1917-01-02 -> today
dofjson --archivo --desde 1980-01-01 --hasta 1980-12-31
dofjson --archivo --pausa 1.0                          # slower (kinder to the server)
```

Output goes to `notas-archivo/` (configurable with `--outdir`), a **local,
never-committed** directory (it is in the repo's `.gitignore`), with the same
per-day filenames the plain command produces:

```
notas-archivo/
  .completados                 # registry of finished days (for resuming)
  2026/
    15072026-notas.json        # index for 2026-07-15 (get_notas, filtered)
    16072026-notas.json
  1980/
    02011980-notas.json
```

The mode is resumable and idempotent: the `.completados` registry records the
finished days, so each run only fetches what is missing. Days that fail with
network errors are *not* marked and get retried on the next run; days the
service does not have (holidays, weekends, very old dates — a 404) *are*
marked so they are not retried forever. "Today" is never marked, so late
additions are picked up by a later run. You can interrupt with Ctrl-C and
resume at any time.

> The full range is ~40,000 days: a long download, meant to be run in parts.
> Start with a bounded range via `--desde/--hasta` if you only need an era.

## Development

```bash
pytest packages/dofjson
```
