Metadata-Version: 2.5
Name: cipo-patent-data
Version: 0.1.0
Summary: Download the Canadian Intellectual Property Office's bulk patent data over its IP Horizons SFTP server, with a verifiable manifest. Unofficial tool; not affiliated with or endorsed by CIPO.
Project-URL: Homepage, https://github.com/maliyuam/cipo-patent-data
Project-URL: Repository, https://github.com/maliyuam/cipo-patent-data
Project-URL: Issues, https://github.com/maliyuam/cipo-patent-data/issues
Project-URL: Changelog, https://github.com/maliyuam/cipo-patent-data/blob/main/CHANGELOG.md
Author-email: Muhammad Aliyu <maliyuam01@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: CIPO,Canada,bulk data,intellectual property,patents,research data,sftp
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: paramiko<6,>=3.4
Provides-Extra: test
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# cipo-patent-data

Download the Canadian Intellectual Property Office's bulk patent data from its **IP Horizons SFTP
server**, with a manifest that records, for every file, its size, SHA-256, retrieval time, source and a
zip integrity test — so a download can be verified and cited later.

> **Unofficial tool.** Not affiliated with or endorsed by the Canadian Intellectual Property Office.
> The MIT licence covers this code only. The data you download, and anything derived from it, is
> governed by CIPO's
> [terms and conditions for the use of Canadian intellectual property data](https://ised-isde.canada.ca/site/canadian-intellectual-property-office/en/terms-and-conditions-use-canadian-intellectual-property-data)
> — read them before redistributing anything.

## Why SFTP

CIPO publishes the same collections over public HTTPS, with no login. The SFTP route adds two things
that matter for research data: a real directory listing (the HTTPS index is not browsable, so the file
list otherwise has to be scraped), and a reference against which to check completeness — archives
served over HTTPS have been found truncated. `compare` uses the listing to tell you whether files you
already hold are the same as the server's, shorter, or a different size.

## Install

```bash
pip install cipo-patent-data
```

Python 3.10 or newer, on Windows, macOS or Linux. The only dependency is `paramiko`.

## 1. Get an account and store the login

SFTP access is issued by CIPO on request. Register through the **"Register for automated downloading
(SFTP)"** form on CIPO's
[IP Horizons: Download intellectual property data](https://ised-isde.canada.ca/site/canadian-intellectual-property-office/en/canadian-intellectual-property-statistics/ip-horizons-download-intellectual-property-data)
page; questions about access go to CIPO's IP data team at `cipoipdata-opicdonneespi@ised-isde.gc.ca`.
CIPO sends a user name and a password. (CIPO's own
[SFTP guide](https://ised-isde.canada.ca/site/canadian-intellectual-property-office/en/automating-file-transfers-ip-horizons-local-drives-using-secure-file-transfer-protocols-sftp)
describes WinSCP; its host key and folder names are out of date — see *Host key* below.)

Create a credentials file from
[`credentials.env.example`](https://github.com/maliyuam/cipo-patent-data/blob/main/credentials.env.example):

| System | Default location |
|---|---|
| Windows | `%APPDATA%\cipo-patents\credentials.env` |
| macOS, Linux | `~/.config/cipo-patents/credentials.env` (or `$XDG_CONFIG_HOME/cipo-patents/`) |

```
CIPO_SFTP_USER=the-user-name-cipo-sent-you
CIPO_SFTP_PASSWORD=
```

- Leave the password empty to be asked for it **once per command** (reconnects reuse it in memory).
- Or keep it out of files entirely: set `CIPO_SFTP_USER` and `CIPO_SFTP_PASSWORD` in the environment.
- Put the file elsewhere with `CIPO_CREDENTIALS=<path>` or `--credentials <path>`.
- On macOS and Linux, `chmod 600` the file.

**Never paste your login into an issue or a log excerpt.** Nothing this tool prints contains the
password. See [SECURITY.md](https://github.com/maliyuam/cipo-patent-data/blob/main/SECURITY.md).

## 2. Choose where data goes

Downloads, the manifest and reports go to `--dest`, which defaults to `$CIPO_DATA_DIR`, or else a
`cipo-data` folder in the current directory:

```bash
export CIPO_DATA_DIR=/data/cipo          # macOS / Linux
setx CIPO_DATA_DIR "E:\research\cipo"     # Windows (new terminals)
```

## 3. First run

Connect **sparingly** (see *Connection limits* below). A good first session:

```bash
cipo-patents dirs                  # one connection: what can this account see?
cipo-patents fetch --limit 3       # one connection: list the patent collections, fetch three files
cipo-patents status                # no connection
```

Then take what you need:

```bash
cipo-patents fetch --directory Patents_ST96_Weekly_Collection
cipo-patents fetch --directory Patent_CSV_2024_10_11 --match PT_main --match PT_interested_party
cipo-patents verify --deep         # no connection: re-hash and CRC-test every archive
```

Directory names change with each production run, so always start from `dirs` rather than a name
written down somewhere — including in CIPO's guide, whose names are out of date. `list`, `plan` and
`fetch` select the patent collections by default; `--all-products` adds trademarks, designs, images
and sequence listings.

## Commands

| Command | Connects | What it does |
|---|---|---|
| `dirs` | yes | Product directories visible to your account |
| `list` | yes | Every file and size → `compare/server_inventory.csv` |
| `plan` | yes | Record the listing in `manifest.csv` without downloading |
| `fetch` | yes | Download what is missing; resumes part files |
| `verify` | no | Re-hash files and test archives (`--deep` checks every CRC) |
| `status` | no | Summary of the manifest |
| `local-inventory` | no | Inventory archives you already hold elsewhere |
| `compare` | no | Server listing vs. what you hold → `compare/comparison.csv` |
| `ingest` | no | File archives you downloaded by hand |

`fetch` options: `--directory` (repeatable), `--match TEXT` (repeatable), `--limit N`, `--recheck`,
`--attempts N`, and `--only-new` to take only files `compare` marked as held nowhere else.

What `fetch` downloads: files not yet held, files marked `failed`, files whose local copy is missing,
and files the server now lists at a different size (`changed`). A file that downloaded in full but
failed its archive test is marked `corrupt` and is **not** downloaded again by a plain `fetch` — CIPO
has been seen serving archives damaged at source, and re-fetching gigabytes on every run would not help.
Retry one deliberately with `fetch --recheck --match <name>`.

### Is this the same data I already have?

```bash
cipo-patents list
cipo-patents local-inventory --root /path/to/existing/cipo/archives
cipo-patents compare
cipo-patents fetch --only-new
```

`compare` puts every file into one of: only on the server; held but **truncated** locally (take the
server copy); same name, different size; same name and size; or held locally but not in the listing
you took. That last group includes older editions, and also any collection your `list` did not cover
(by default `list` covers the patent collections only; use `list --all-products` to compare everything).

`local-inventory` never opens cloud-storage placeholders (OneDrive, iCloud Drive and similar), because
opening one downloads it: they are reported as `cloud-only` with their size. Placeholders are detected
on Windows and macOS; elsewhere they cannot be detected and would be read.

### Files downloaded by hand

If you use WinSCP or FileZilla instead, run `cipo-patents list` once, save files anywhere, then:

```bash
cipo-patents ingest                # scans <dest>/raw and ~/Downloads, recursively
cipo-patents ingest --from /path/to/downloads --move
```

Each file's size is checked against the server listing (a short download is reported and not filed),
the zip is tested, the file is hashed and copied into `raw/<directory>/`, and the row is marked
`source = manual`. A file already saved in the right folder is adopted in place (renamed to the server's
spelling if only the letter case differs).

With `--move`, an original is deleted only when an identical copy is filed: either the copy just made
matched the original's SHA-256, or the filed copy was re-hashed and matched. If the filed copy has been
damaged, the good original is filed over it instead. Every deletion is printed.

## Host key

Before sending your login the tool checks the server's host key and refuses to connect if it does not
match the pinned value.

> **The fingerprint in CIPO's guide is out of date.** Guide v2.0 prints
> `ssh-ed25519 256 wOX1WxRWVdy096BQNLi+T2VXtRLIvs1aS160SR3CguA=`; when last measured (2026-09-16) the
> server offered no ed25519 key at all, only a 2048-bit RSA key:
>
> `ssh-rsa 2048 SHA256:v5YTto75n4lEn8uflIUH0aZfOxpUq0dlidXT/qpif2E`
>
> That RSA key is what this tool pins. It was observed, not confirmed by CIPO.

If CIPO changes the key you will get a mismatch error and no login is sent. Confirm the new fingerprint
with CIPO, then set it:

```
CIPO_SFTP_HOSTKEY=ssh-rsa 2048 SHA256:<new fingerprint>
```

## Connection limits

The server refuses an address for a while — about 40 minutes when observed — after several
connections in quick succession. Each command opens **one** session. `fetch` reconnects only when a
transfer drops, waits between attempts (30 s, then longer), and stops after eight reconnects in one
run; running the same command again later resumes where it stopped. Avoid scripting loops of `dirs`
or `list`.

Large transfers used to drop during SSH key renegotiation; the tool suppresses rekeying for the
session and reads ahead in 64 MB windows, which keeps multi-gigabyte files moving.

## Layout

```
<dest>/
  raw/<server directory>/...   downloaded archives
      *.part, *.part.json      an interrupted download and the server version it belongs to
  compare/                     server_inventory.csv, local_inventory.csv, comparison.csv
  manifest.csv                 one row per server file (see below)
  manifest.pending.csv         only if a save could not replace a locked manifest; read automatically
  .cipo-patents.lock           present while a command is running
```

`manifest.csv` columns: `relpath`, `directory`, `filename`, `product`, `cadence`, `remote_size`,
`remote_mtime`, `status`, `source`, `bytes`, `sha256`, `retrieved_at_utc`, `archive_test`,
`tested_at_utc`, `note`. Statuses: `planned`, `downloaded`, `corrupt`, `failed`, `missing`, `changed`,
`withdrawn`. The manifest never contains credentials.

`changed` covers both a local copy that no longer matches its record and a newer version listed on the
server; `verify` keeps the second visible until `fetch` has taken the new version, and it restores a row
to `downloaded` only after re-hashing it (never under `--no-rehash`).

**Before sharing or depositing the folder**, leave out `compare/` (its reports contain absolute paths
from your machine), any `*.part` and `*.part.json` files (incomplete downloads), and the lock file.
Keep the credentials file out of the data folder altogether.

## Environment variables

| Variable | Meaning |
|---|---|
| `CIPO_DATA_DIR` | Default for `--dest` |
| `CIPO_CREDENTIALS` | Path of the credentials file |
| `CIPO_SFTP_USER`, `CIPO_SFTP_PASSWORD` | Login, overriding the file |
| `CIPO_SFTP_HOST`, `CIPO_SFTP_PORT` | Server, if CIPO moves it |
| `CIPO_SFTP_HOSTKEY` | Pinned host key, after confirming a change with CIPO |
| `CIPO_DEBUG` | Set to any value to let paramiko's own diagnostic logging through (it can include local paths) |

## Development

```bash
git clone https://github.com/maliyuam/cipo-patent-data
cd cipo-patent-data
pip install -e ".[test]"
pytest
```

The test suite runs entirely offline. Please do not add tests that connect to CIPO's server.

## Citation

See [CITATION.cff](https://github.com/maliyuam/cipo-patent-data/blob/main/CITATION.cff). Cite CIPO's
data separately, according to CIPO's terms.

## Licence

MIT, for the code. See [LICENSE](https://github.com/maliyuam/cipo-patent-data/blob/main/LICENSE).
