Metadata-Version: 2.4
Name: warc2zip
Version: 1.0.0
Summary: Convert WARC files into zip-of-files plus metadata
License-Expression: Apache-2.0
Project-URL: Repository, https://github.com/commoncrawl/warc2zip
Project-URL: Issues, https://github.com/commoncrawl/warc2zip/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fsspec[http]
Requires-Dist: tqdm>=4.67.3
Requires-Dist: warcio[s3]>=1.8.1
Provides-Extra: dev
Requires-Dist: ruff==0.14.3; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# warc2zip

`warc2zip` converts WARC web archive files into zip archives, while preserving 100% of the metadata.

The goal of warc2zip is to facilitate low-code and no-code usage of web archives. A researcher
can make a selection of WARCs, perhaps from an Archive-It collection, perhaps a Browsertrix
collection, perhaps a repackage of Common Crawl containing just webpages labeled as being in
Swahili. `warc2zip` then converts these WARCs into zip files containing the web capture payloads,
with metadata stored as csv (spreadsheet) files.

Each response record's payload is stored as a individual file with a proper extension, derived from its Content-Type.
Metadata (both WARC and http) from the request, response, and metadata records are  written to CSV (spreadsheet) files.

> [!WARNING]
> **Feedback is welcome**: this project is in early development. Feel free to open an issue or submit
> a pull request if you have suggestions, bug reports, or feature requests. See [WARC examples](#warc-examples-for-testing) for testing below.

## Installation

> [!TIP]
> We recommend you to use a virtual environment (e.g. `uv` or `conda`) to avoid conflicts with other Python packages.

```bash
pip install .
```

By default, pip will install remote access tools, namely `fsspec` configured to talk to https, s3, and Internet Archive (`ia://`) remote files.

## Usage

```bash
warc2zip <path/to/file.warc.gz>
```

Input can be a local path or a remote URI (S3, HTTP, an Internet Archive item, etc.):

```bash
warc2zip s3://commoncrawl/crawl-data/.../CC-MAIN-....warc.gz
warc2zip https://data.commoncrawl.org/crawl-data/.../CC-MAIN-....warc.gz
warc2zip ia://EOT24PRE-20240926175758-crawl808/EOT24PRE-20240926175758-00032.warc.gz
```

`ia://<identifier>/<filename>` names a file in an [archive.org](https://archive.org) item; it is read from
`https://archive.org/download/<identifier>/<filename>`, so the two spellings are interchangeable. Public items
need no account. For a restricted item, log in once with the `internetarchive` package — `pip install
internetarchive && ia configure` writes `~/.config/internetarchive/ia.ini`, which `warc2zip` reads the way `ia`
does (`$IA_CONFIG_FILE` first) — or set `IA_ACCESS_KEY_ID` and `IA_SECRET_ACCESS_KEY`. A refused item says which
of those it tried.

**Note**: Please use s3 inside of AWS and https outside.

**Note**: `s3://commoncrawl` does **not** allow anonymous access — requests must be signed with credentials from any AWS account. To fetch Common Crawl data without an AWS account, use the HTTPS endpoint (`https://data.commoncrawl.org/...`) instead.

### Options

| Flag                      | Description                                                                            | Default                                 |
|---------------------------|----------------------------------------------------------------------------------------|-----------------------------------------|
| `input_file`              | Path or URI to a `.warc.gz` file: local, `s3://`, `http(s)://` or `ia://<item>/<file>` (positional, required) |                                         |
| `--output`                | Path to the output zip file                                                            | `{basename}_{hex}.zip` in the current directory, same hex and `_partial` rule as the root directory inside |
| `--dry-run`               | Print summary without creating output. The scan always stops after at most 10 capture records, so it never streams the whole file; a lower `--limit` is respected |                                         |
| `--limit <N>`             | Limit to N capture records, with their full set of associated request/metadata records | No limit, all records are processed     |
| `--format {flat,sidecar}` | Output format (see [Output Formats](#output-formats) below)                            | `flat`                                  |
| `--metadata-only`         | Write every CSV, manifest and sidecar but no payload files                             | Off, payloads are written               |
| `--select <csv>`          | Convert only the captures listed in a `manifest.csv` from an earlier run on the same WARC (see [Building and downloading a subset](#building-and-downloading-a-subset)) | Off, every capture is converted |

### Small Examples

Preview records without writing anything:

```bash
warc2zip archive.warc.gz --dry-run
```

Specify output path:

```bash
warc2zip archive.warc.gz --output result.zip
```

Process a remote WARC file from S3:

```bash
warc2zip s3://bucket/path/archive.warc.gz --output local-result.zip
```

Sample the first 10 capture records (response + associated request/metadata) — useful for quickly inspecting a large remote WARC without streaming the whole file:

```bash
warc2zip s3://bucket/path/archive.warc.gz --limit 10 --output sample.zip
```

Extract with sidecar format (per-file metadata, grouped by domain):

```bash
warc2zip archive.warc.gz --format sidecar --output result.zip
```

Take the metadata and leave the payloads behind — for massaging the metadata before deciding which captures you actually want:

```bash
warc2zip archive.warc.gz --metadata-only --output metadata.zip
```

Every CSV, manifest, warcinfo file and sidecar is written exactly as it would be in a full run; only the payload files are omitted. Note this saves **output size, not transfer**: the WARC is still streamed from end to end, because counting and describing the records means reading them. `--limit` is the flag that shortens the read.

## Reading the run summary

Every run ends with a summary line, a table of every record read, and, when something was left out, a warning on stderr. This is a Common Crawl `crawldiagnostics/` file:

```
Created CC-MAIN-20260807101845-20260807131845-00000.zip: 3639 responses, 150 revisits, 3789 requests, 3789 metadata records
Record types:
  warcinfo     1
  response  3639
  revisit    150
  request   3789
  metadata  3789
```

### The `Record types` table

One row per `WARC-Type` found in the file, so the table always adds up to the number of records read. warc2zip extracts five types, listed first; everything else is read and discarded, and flagged `(not extracted)`. The table is the only place such a record is visible at all, so if a WARC seems to be "missing" captures, look here first.

WARC 1.1 defines eight record types. Seven occur in crawl WARCs; the eighth, `conversion`, holds a transformed copy of a response (Common Crawl's WET files — the extracted plain text — are made of nothing else) and does not appear in the `warc/` family.

| `WARC-Type`    | What it is                                                                                                                                                                                              | Typical producers                                                                                             | warc2zip                                                                                     |
|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|
| `warcinfo`     | Describes the file itself: crawler software, operator, robots policy, `WARC-Filename`. Usually the first record; a concatenated WARC has several.                                                        | Everyone. ARC's `filedesc://` record is read as one.                                                          | Extracted → `warcinfo.warc`, `warcinfo.warc-fields`, `warcinfo.csv`; names the root directory |
| `response`     | A complete HTTP response — status line, headers and body — for `WARC-Target-URI`. The anchor of a capture.                                                                                              | Everyone. ARC records are read as this type.                                                                  | Extracted → the payload file and the `manifest.csv` row                                      |
| `request`      | The HTTP request that produced a response. Linked to it by `WARC-Concurrent-To` (on the response, in CC and wget output).                                                                               | CC, Heritrix, wget, Browsertrix                                                                               | Extracted → `request_*.csv`, `.request.*` sidecars                                           |
| `metadata`     | Crawler-side facts about a capture, as `application/warc-fields`: `fetchTimeMs`, detected charset and languages, outlinks. `WARC-Concurrent-To` names the record it describes.                          | CC, Heritrix. wget writes one for its own log; Browsertrix does not write them                                | Extracted → `metadata.csv` (body flattened), `metadata_multi.csv` (body raw), `.metadata.*` sidecars |
| `revisit`      | A capture whose content was **not stored** because it duplicates an earlier one. Two profiles: `server-not-modified` (a `304`, headers only) and `identical-payload-digest` (fetched, same hash, body discarded). `WARC-Refers-To-Target-URI` / `-Date` say which capture holds the bytes. | CC `crawldiagnostics/`, Heritrix, Browsertrix, wget with `--warc-dedup`                                       | Extracted → a `manifest.csv` row and every header CSV, but no payload file; the row's `warc_refers_to_*` columns say which capture holds the content |
| `resource`     | A block that *is* the content, with no HTTP transaction around it: DNS lookups (`dns:` URIs, `text/dns`), `ftp://` fetches, screenshots (`urn:screenshot:`), page text, crawler logs.                   | Heritrix (DNS), wget (FTP, its own `wget.log`), Browsertrix and warcprox (screenshots, `urn:pageinfo:`)       | Not extracted. Leaves no other trace — only the table shows it                               |
| `continuation` | Segment 2..N of a record too large for one file. The first segment is a normal `response` with `WARC-Segment-Number: 1`; the rest carry `WARC-Segment-Origin-ID`.                                       | Heritrix, when configured to segment. **Never Common Crawl**, which truncates at 1 MiB and sets `WARC-Truncated: length` instead | Not extracted. **Caveat:** the first segment *is* a response, so its payload file is written holding only that segment |

Where a producer's habits matter: a `resource` record — how Heritrix stores DNS lookups, and how Browsertrix stores screenshots and page text — is a real capture with a real payload, but since it has no HTTP layer it is not a `response` and is skipped. The pre-2009 ARC format has no record types at all; every ARC record, DNS lookups included, is read as a `response` (see the EOT-2004 examples below), which is why ARC `dns:` records *are* extracted while WARC-era `resource` DNS records are not.

### The warning

A **capture** is a response (or revisit) with its request and metadata records, and a `manifest.csv` row and the rows in every header CSV all hang off that anchor record. When the anchor is missing or of a type that is not extracted, the whole capture is dropped — **not just the payload** — and a stderr warning accounts for it:

```
Warning: skipped N capture(s) that have no response record, along with their R request and M metadata records
```

Causes:

- **A truncated file**, or a stream cut off mid-capture — a request whose response never arrived.
- **A `resource` record with a metadata record pointing at it.** The metadata is counted as dropped; the resource itself only shows in the table.
- **A producer that writes no metadata records** (wget, Browsertrix). Then only request records can be counted, so the capture figure comes from them.

Warnings never change the exit status; warc2zip exits 1 only when a CSV row could not be written.

```
warning: <input>: attempt 1/9 failed (503, message='Service Unavailable', ...), retrying in 2.6 s
```

A transient failure while reading a remote input — throttling, a 5xx, a connection dropped
mid-block. The read resumes exactly where it stopped, so nothing is duplicated or lost. Up to
8 retries with exponential backoff (capped at 60 s, `Retry-After` honoured); after that the
error is raised. A pipe (`-` on stdin) cannot be rewound, so it is not retried.

## Output Formats

All files are placed under a unique root directory inside the zip to prevent collisions when extracting multiple archives into the same folder. The directory name is derived from the WARC-Filename header (in the `warcinfo` record), the current timestamp, and a random suffix: `{crawl_name}_{YYYYMMDDTHHMMSS}_{hex}`. Without `--output`, the zip carries the same hex (`archive_{hex}.zip`), so same-named inputs such as Common Crawl's `warc/`, `crawldiagnostics/` and `robotstxt/` files never overwrite each other's zip. A `--limit` run appends `_partial` to both names (`archive_{hex}_partial.zip`).

This testing release of the software supports 2 output formats: flat and sidecar.
Flat puts the metadata into a small number of large files, and sidecar instead
creates a lot of metadata files, one for each payload.

To see real output before installing anything, download one of the ready-made zips
listed under [WARC examples for testing](#warc-examples-for-testing): each example WARC
sits next to its `--format flat` conversion, its `--format sidecar` conversion and its `--metadata-only` conversion.

### Flat format (`--format flat`, default)

Counter-named payload files with global CSV/JSONL metadata. Optimized for bulk analysis.

```
FOO.zip
  CC-MAIN-20251215005813-20251215035813-00995_20260330T143022_a1b2/
    1000000.html              # one per response record, extension from Content-Type
    1000001.pdf
    1000002.unk               # fallback for unknown mime-types
    ...
    warcinfo.warc             # the warcinfo record's WARC headers, raw
    warcinfo.warc-fields      # the warcinfo record's body, raw
    warcinfo.csv              # the same, parsed and greppable
    warcinfo_multi.csv
    manifest.jsonl            # one JSON line per response (mime-type, status, URI, etc.)
    manifest.csv              # the same, one wide row per response
    response_warc_headers.csv # filename, header_name, header_value
    response_warc_headers_multi.csv
    response_http_headers.csv
    response_http_headers_multi.csv
    request_warc_headers.csv
    request_warc_headers_multi.csv
    request_http_headers.csv
    request_http_headers_multi.csv
    metadata.csv
    metadata_multi.csv
```

- **Payload files**: raw response bodies named `{counter}.{ext}`, where the extension is derived from the HTTP Content-Type header via `mimetypes.guess_extension()` (with overrides for common types like `text/html` → `.html`)
- **Denormalized CSVs** (`*_headers.csv`, `metadata.csv`, `warcinfo.csv`): multiple rows per file — columns: `filename, header_name, header_value`. Header names are normalized to lowercase with `-` replaced by `_`.
- **Multiline CSVs** (`*_multi.csv`): one row per file — columns: `filename, headers` (headers as a multiline string)
- **`manifest.csv`**: the wide mirror of `manifest.jsonl` — one row per response, one column per key. Both are written from the same entries, so they cannot drift.
- **`warcinfo.*`**: crawl-level provenance (`isPartOf`, `publisher`, `software`, `hostname`, `conformsTo`, …). The `filename` column carries the synthetic key `warcinfo`, since the record belongs to no payload file; a concatenated WARC with several `warcinfo` records numbers the extras `warcinfo.1`, `warcinfo.2`, ….

### Sidecar format (`--format sidecar`)

Captures grouped by domain (from `WARC-Target-URI`), with per-file metadata sidecars alongside each payload. Optimized for browsing individual captures. Global CSVs and manifest.jsonl are still generated.

```
FOO.zip
  CC-MAIN-20251215005813-20251215035813-00995_20260330T143022_a1b2/
    example.com/
      1000000.html                       # response payload
      1000000.html.request.warc          # request WARC headers
      1000000.html.request.http          # request HTTP headers (with request line)
      1000000.html.request.json          # request HTTP body (often empty for GET)
      1000000.html.response.warc         # response WARC headers
      1000000.html.response.http         # response HTTP headers (with status line)
      1000000.html.metadata.warc         # metadata WARC headers
      1000000.html.metadata.warc-fields  # metadata body (application/warc-fields)
    other.org/
      1000001.pdf
      1000001.pdf.response.warc
      1000001.pdf.response.http
      ...
    warcinfo.warc
    warcinfo.warc-fields
    manifest.jsonl
    manifest.csv
    response_warc_headers.csv
    ...
```

- **Domain directories**: extracted from `WARC-Target-URI` (e.g. `https://example.com/page` → `example.com/`). Captures without a target URI go under `unknown/`.
- **Sidecar files**: named by appending a suffix to the payload filename. Headers are written in raw `Name: value` format (not normalized). Multiple metadata records for the same capture are merged, separated by blank lines.
- **Missing records are skipped**: if a capture has no request or metadata record, the corresponding sidecar files are simply not created.

## Metadata Examples

### manifest.jsonl

One JSON line per response record:

```json
{"filename": "1000000.html", "warc_record_id": "<urn:uuid:12345678-abcd-...>", "warc_target_uri": "https://example.com/page", "warc_date": "2025-12-15T00:58:13Z", "http_status_code": "200", "detected_mime_type": "text/html", "content_type_header": "text/html; charset=UTF-8", "payload_size": 34521}
{"filename": "1000001.pdf", "warc_record_id": "<urn:uuid:87654321-dcba-...>", "warc_target_uri": "https://example.com/doc.pdf", "warc_date": "2025-12-15T00:58:14Z", "http_status_code": "200", "detected_mime_type": "application/pdf", "content_type_header": "application/pdf", "payload_size": 102400}
```

### response_warc_headers.csv (denormalized)

One row per header per file:

```csv
"filename","header_name","header_value"
"1000000.html","warc_type","response"
"1000000.html","warc_date","2025-12-15T00:58:13Z"
"1000000.html","warc_target_uri","https://example.com/page"
"1000000.html","warc_record_id","<urn:uuid:12345678-abcd-...>"
"1000000.html","content_length","34521"
"1000000.html","warc_protocol","h2"
"1000000.html","warc_protocol","tls/1.3"
```

Note the repeated `warc_protocol` rows: this is where the capture's real protocol lives. The HTTP status line in Common Crawl WARCs claims `HTTP/1.1` even for an h2 capture, which is why `status_code` below carries only the code and never the status line.

### response_warc_headers_multi.csv (multiline)

One row per file, all headers in a single multiline string. Header names are normalized to lowercase with `-` replaced by `_`:

```csv
"filename","headers"
"1000000.html","warc_type: response
warc_date: 2025-12-15T00:58:13Z
warc_target_uri: https://example.com/page
warc_record_id: <urn:uuid:12345678-abcd-...>
content_length: 34521"
```

### response_http_headers.csv (denormalized)

Each file's block opens with a synthetic `status_code` row, so a status can be grepped for directly:

```csv
"filename","header_name","header_value"
"1000000.html","status_code","200"
"1000000.html","content_type","text/html; charset=UTF-8"
"1000000.html","content_length","34521"
"1000000.html","server","nginx/1.18.0"
"1000000.html","date","Mon, 15 Dec 2025 00:58:13 GMT"
```

```bash
unzip -p FOO.zip '*/response_http_headers.csv' | grep '"status_code","403"'
```

`status_code` is a pseudo-header, not a header off the wire. A real response header literally named `Status-Code` would normalize to the same name — no such header exists in practice, but the collision is worth knowing about.

### request_warc_headers.csv (denormalized)

```csv
"filename","header_name","header_value"
"1000000.html","warc_type","request"
"1000000.html","warc_date","2025-12-15T00:58:13Z"
"1000000.html","warc_target_uri","https://example.com/page"
"1000000.html","warc_record_id","<urn:uuid:abcdef01-2345-...>"
```

### request_http_headers.csv (denormalized)

The request line is split into `request_method` / `request_target` pseudo-headers, the same way the status line becomes `status_code`:

```csv
"filename","header_name","header_value"
"1000000.html","request_method","GET"
"1000000.html","request_target","/"
"1000000.html","user_agent","CCBot/2.0 (https://commoncrawl.org/faq/)"
"1000000.html","accept_language","en-US,en;q=0.5"
"1000000.html","accept_encoding","zstd, br, gzip"
```

### metadata.csv (denormalized)

The metadata record's `application/warc-fields` body is shallow-flattened: one row per field, named `_body.<field>`. One level only — a field whose value is JSON (Common Crawl's `languages-cld2`) stays a single opaque value, because the list nested inside it does not flatten into columns usefully.

```csv
"filename","header_name","header_value"
"1000000.html","warc_type","metadata"
"1000000.html","warc_date","2025-12-15T00:58:13Z"
"1000000.html","warc_concurrent_to","<urn:uuid:12345678-abcd-...>"
"1000000.html","_body.fetchtimems","1044"
"1000000.html","_body.charset_detected","UTF-8"
"1000000.html","_body.languages_cld2","{""reliable"":false,""text-bytes"":29,""languages"":[{""code"":""zh"",""name"":""Chinese""}]}"
```

`metadata_multi.csv` keeps the body as one verbatim `_body` block instead, so the raw wire bytes survive in flat format too. A body that is not valid warc-fields falls back to a single raw `_body` row in both files.

### manifest.csv

The wide mirror of `manifest.jsonl` — one row per capture (response or revisit), one column per key:

```csv
"filename","warc_type","warc_record_id","warc_target_uri","warc_date","http_status_code","detected_mime_type","content_type_header","payload_size","warc_refers_to_target_uri","warc_refers_to_date","warc_filename","source_uri","warc_record_offset","warc_record_length"
"1000000.html","response","<urn:uuid:12345678-abcd-...>","https://example.com/page","2025-12-15T00:58:13Z","200","text/html","text/html; charset=UTF-8","34521","","","CC-MAIN-20251215005813-20251215035813-00995.warc.gz","https://data.commoncrawl.org/crawl-data/CC-MAIN-2025-51/segments/.../CC-MAIN-...warc.gz","1062","3585"
"1000038.revisit","revisit","<urn:uuid:9abcdef0-1234-...>","http://avt-studio.de/index.htm","2026-08-07T11:19:40Z","304","application/octet-stream","","0","http://avt-studio.de/index.htm","2026-05-11T12:10:15Z","CC-MAIN-20260807101845-20260807131845-00000.warc.gz","https://data.commoncrawl.org/crawl-data/.../CC-MAIN-...warc.gz","264601","597"
```

A `revisit` row names no file in the zip (its `filename` is a synthetic join key): the capture was a `304 Not Modified`, so its content lives in an earlier capture — `warc_refers_to_target_uri` + `warc_refers_to_date` say which one, resolvable through the CC index. Filter `warc_type == "response"` for exactly the files that exist in the zip.

The last four columns are what make a row re-fetchable on its own, and they are repeated on every row on purpose — no join against another file, no knowledge of the zip they came from:

| Column                                      | Meaning                                                                                                                                                                               |
|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `warc_filename`                             | What the WARC **calls itself** (the `warcinfo` record's `WARC-Filename`), falling back to the input's basename. A label — for Common Crawl it is a bare basename, not a fetchable path. |
| `source_uri`                                | Where warc2zip **read the file from** — the `input_file` argument, verbatim. **This is the fetch target.**                                                                            |
| `warc_record_offset` / `warc_record_length` | Byte range of the record. The same triple a CDX index carries.                                                                                                                        |

The offsets are positions in the file named by `source_uri` — that is the only file they are guaranteed to address. `warc_filename` may name a *different* file: a derived WARC (an extract, or the output of `tools/warc_limit.py`) copies the original `warcinfo` record, so it keeps advertising the original WARC's name while its byte offsets refer to the derived file. Use `source_uri` to fetch, `warc_filename` to say where the records originated. See [Building and downloading a subset](#building-and-downloading-a-subset).

### warcinfo.csv

Crawl-level provenance, with the record body flattened the same way as `metadata.csv`:

```csv
"filename","header_name","header_value"
"warcinfo","source_uri","https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-25/segments/.../CC-MAIN-...warc.gz"
"warcinfo","warc_record_offset","0"
"warcinfo","warc_record_length","519"
"warcinfo","warc_type","warcinfo"
"warcinfo","warc_filename","CC-MAIN-20260618163205-20260618193205-00999.warc.gz"
"warcinfo","_body.ispartof","CC-MAIN-2026-25"
"warcinfo","_body.publisher","Common Crawl"
"warcinfo","_body.software","Apache Nutch 1.21 (modified, https://github.com/commoncrawl/nutch/)"
"warcinfo","_body.conformsto","https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/"
```

`warcinfo.warc` and `warcinfo.warc-fields` hold the same record as raw wire bytes.

`source_uri` is the input this zip was converted from. It is also a column on every `manifest.csv` row; the copy here is the crawl-level one, written even when the WARC carries no `warcinfo` record at all, so the provenance is never lost.

Both `source_uri` and the `warc_record_*` fields are pseudo-headers — computed while streaming, not read off the wire. A real warcinfo header named `Source-URI` would normalize to the same name; none exists in practice, but the collision is worth knowing about, same as for `status_code`.

`warc_record_offset` and `warc_record_length` are not headers off the wire; they are computed while streaming, and appear for every record type — in `manifest.csv` for responses, and as rows in `response_warc_headers.csv`, `request_warc_headers.csv`, `metadata.csv` and `warcinfo.csv`.

## Building and downloading a subset

`warc_filename` + `warc_record_offset` + `warc_record_length` make every `manifest.csv` row enough to fetch that capture again on its own. Each record is its own gzip member, so those bytes are a standalone WARC — one HTTP range request per record, no reprocessing of the source file.

The intended workflow is: convert once with `--metadata-only` (a few tens of KB instead of gigabytes), filter the CSV however you like, then convert the same WARC again with `--select` to get a zip with only the captures you kept.

### Get metadata only — no payloads
```bash
warc2zip 'https://data.commoncrawl.org/crawl-data/.../CC-MAIN-....warc.gz' --metadata-only --output meta.zip
unzip -p meta.zip '*/manifest.csv' > manifest.csv
```

### Filter it with whatever you already use

You can open the CSV in Excel, LibreOffice, Python, R, etc. and filter it there, then save it back to CSV. 
You can skip to the [next step](#convert-what-you-kept).  

To use command-line tools like `awk`, `csvkit`, `miller`, `xsv`, `q`, etc, read the next few lines. 
 
For example to get all records with http response code `200` from the column `http_status_code`:

1. List the column names:

    ```bash
    head -1 manifest.csv | tr ',' '\n' | tr -d '"\r' | nl
    ```

2. Filter by column name and value. Here some examples with `awk` and [miller](https://miller.readthedocs.io/en/latest/).

    with `awk`:
    ```bash
    awk -F'","' -v col="http_status_code" -v val="200" '
    NR==1 { for (i=1; i<=NF; i++) { h=$i; gsub(/["\r]/,"",h); if (h==col) c=i } print; next }
    { v=$c; gsub(/["\r]/,"",v) } v==val
    ' manifest.csv > subset.csv
    ```
    or with [miller](https://miller.readthedocs.io/en/latest/):
    ```
    mlr --csv filter '$http_status_code == "200"' manifest.csv > subset.csv
    ```

### Convert what you kept
```bash
warc2zip 'https://data.commoncrawl.org/crawl-data/.../CC-MAIN-....warc.gz' --select subset.csv --output subset.zip
```

The zip has the usual layout with only the listed captures, each one complete: payload, response, request and metadata records, plus the WARC's `warcinfo`. The WARC is read in full again, as for `--metadata-only`. Rows are matched by `warc_record_offset` (checked against `warc_record_id` when the row has one), so a `manifest.csv` from a selected zip works with `--select` too.

> [!TIP]
> Select from the same WARC the manifest came from. It does not have to be at the same location: a local copy of an `s3://` file works, because rows are matched by file name. Rows that name another WARC are reported and skipped; run once per WARC for a subset that spans several.

Offsets stay valid under `--limit`: limiting only stops the read early, it never rewrites them.

## WARC examples for testing

We prepared some smaller (~1GBytes or less) and interesting WARC files for testing: US Federal government websites, homepages, etc.
These files are in a [Huggingface bucket](https://huggingface.co/buckets/commoncrawl/warc2zip-examples) and the `warc2zip` commands
below read directly from that bucket. 
Next to each WARC the bucket also holds the zips `warc2zip` made from it: one per output format
(`--format flat` and `--format sidecar`) and a `--metadata-only` zip (every CSV, no payload files),
so you can look at the output without running anything.
The commands below use `--format flat` ... you can also try `--format sidecar`

Details:

- Example WARC from CC-MAIN-2026-25 with 500 records (response, request and metadata, 13 MBytes) [500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz?download=true)

  - make the zip
```
  warc2zip 'https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz?download=true' --format flat
```
  - or download the zip we made: [500_RECORDS-REPACKAGE-CC-MAIN-2026-30.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.zip?download=true) (13 MBytes), sidecar format: [500_RECORDS-REPACKAGE-CC-MAIN-2026-30.sidecar.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.sidecar.zip?download=true) (14 MBytes), metadata only: [500_RECORDS-REPACKAGE-CC-MAIN-2026-30.metadata-only.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.metadata-only.zip?download=true) (0.8 MBytes)
  - here is the warcinfo

```
  isPartOf: CC-MAIN-2026-25
  publisher: Common Crawl
  description: Wide crawl of the web for June 2026
  operator: Common Crawl Admin (info@commoncrawl.org)
  hostname: ip-10-67-67-233
  software: Apache Nutch 1.21 (modified, https://github.com/commoncrawl/nutch/)
  robots: checked via crawler-commons 1.7-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
  format: WARC File Format 1.1
  conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/
  ```

- Homepages extracted from CC-MAIN-2026-21 (response records only, 1 GByte) [HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz?download=true)
  - make the zip, note the limit
```
warc2zip 'https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz?download=true' --format flat --limit 1000
```
  - or download the zip we made, which is the full conversion without the limit: [HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.zip?download=true) (1 GByte), sidecar format: [HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.sidecar.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.sidecar.zip?download=true) (1.1 GBytes), metadata only: [HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.metadata-only.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.metadata-only.zip?download=true) (35 MBytes)
  - here is the warcinfo
```
  software: pypi_cdx_toolkit/0.9.40.dev89+g53a7ef76c
  isPartOf: CC-MAIN-2026-21
  description: Repackage of CC-MAIN-2026-21 containing only response records of homepages
  format: WARC file version 1.0
  creator: Common Crawl Foundation <https://commoncrawl.org>
  operator: Malte Ostendorff <mailto:malte@commoncrawl.org>
  ```
- URLs of federal institutions (response records only, 1/2 GByte), as part of the [End Of Term Archive](https://eotarchive.org/) project: [IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz?download=true)
  - make the zip, note the limit
```
warc2zip 'https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz?download=true' --format flat --limit 1000
  ```
  - or download the zip we made, which is the full conversion without the limit: [IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.zip?download=true) (434 MBytes), sidecar format: [IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.sidecar.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.sidecar.zip?download=true) (494 MBytes), metadata only: [IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.metadata-only.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.metadata-only.zip?download=true) (36 MBytes)
  - here is the warcinfo
```
  software: pypi_cdx_toolkit/0.9.40.dev91+ga04800ea0
  isPartOf: CC-MAIN-2025-13
  description: Repackage of CC-MAIN-2025-13 containing only response records of US federal government hosts
  format: WARC file version 1.0
  creator: Common Crawl Foundation <https://commoncrawl.org>
  operator: Malte Ostendorff <mailto:malte@commoncrawl.org>
  ```

## Many more WARC examples for testing

Every archive in this section and the previous one is a test case in `tests/test_readme_warcs.py`.
They need the network, so a plain `pytest` skips them: `pytest -m integration` converts the first
20 captures of each (the "integration" CI job), `pytest -m long` converts the whole files (also
available as the manual "Long tests" workflow under Actions).

### Common Crawl style repackaged WARCs (intended for testing)

#### HuggingFace: 
- https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz?download=true
- https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz?download=true
- https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz?download=true

#### AWS:
**Prefixes**: https://data.commoncrawl.org/ or `s3://commoncrawl/`:
- /projects/warc2zip-examples/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz (13 MBytes)
- /projects/warc2zip-examples/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz (1 GByte)
- /projects/warc2zip-examples/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz (1/2 GByte)

### Normal Common Crawl CC-MAIN WARCs

**Prefixes**: https://data.commoncrawl.org/ or `s3://commoncrawl/`: 
- crawl-data/CC-MAIN-2026-34/segments/1786091384908.68/warc/CC-MAIN-20260807101845-20260807131845-00000.warc.gz
- crawl-data/CC-MAIN-2026-34/segments/1786091384908.68/crawldiagnostics/CC-MAIN-20260807101845-20260807131845-00000.warc.gz
- crawl-data/CC-MAIN-2026-34/segments/1786091384908.68/robotstxt/CC-MAIN-20260807101845-20260807131845-00000.warc.gz

### End Of Term Archive (https://eotarchive.org/data/)

**Prefixes**: https://eotarchive.s3.amazonaws.com/ or `s3://eotarchive/`: 

#### Heretrix/IA style WARCs from EOT 2024

- crawl-data/EOT-2024/segments/IA-000/warc/EOT24PRE-20240926172119-crawl804_EOT24PRE-20240926172119-00000.warc.gz

#### Nutch/CCF style WARCs from EOT 2024

- crawl-data/EOT-2024/segments/CC-000/warc/EOT-2024-REPACKAGE-CC-MAIN-2024-42-GOV-000000-001.warc.gz

#### Browsertrix style WARCs, EOT 2024

- crawl-data/EOT-2024/segments/WR-000/warc/EOT24WR-0015_20250114215650265-8c53efcc-e2d-0_eot-http-energy-gov-eere-office-energy-efficiency-renewable-energy-manual-20250114215335-8c53efcc-e2d-20250114215647018-0.warc.gz
- crawl-data/EOT-2024/segments/WR-000/warc/EOT24WR-0015_20250114215650265-8c53efcc-e2d-0_eot-http-energy-gov-eere-office-energy-efficiency-renewable-energy-manual-20250114215335-8c53efcc-e2d-screenshots-20250114215649547.warc.gz
- crawl-data/EOT-2024/segments/WR-000/warc/EOT24WR-0015_20250114215650265-8c53efcc-e2d-0_eot-http-energy-gov-eere-office-energy-efficiency-renewable-energy-manual-20250114215335-8c53efcc-e2d-text-20250114215649747.warc.gz

#### ArchiveTeam style megaWARCs, EOT 2024 (warning: 10 gigabytes)

- crawl-data/EOT-2024/segments/AT-000/warc/archiveteam_usgovernment_20250131232111_96ad506d_usgovernment_20250131232111_96ad506d.1738361595.megawarc.warc.gz

#### Heretrix-style ARCs from EOT 2004 (arc is the predecessor to warc)

- crawl-data/EOT-2004/segments/NARA-000/warc/NARA-PEOT-2004-20041014205819-00000-crawling009-c_NARA-PEOT-2004-20041014205819-00000-crawling009.archive.org.arc.gz

### Internet Archive items (`ia://`)

A public archive.org item holding an EOT 2024 Heritrix WARC (1.6 GBytes). The two spellings read the same bytes;
the `ia://` one also works for restricted items once you are logged in (see [Usage](#usage)):

- https://archive.org/download/EOT24PRE-20240926175758-crawl808/EOT24PRE-20240926175758-00032.warc.gz
- ia://EOT24PRE-20240926175758-crawl808/EOT24PRE-20240926175758-00032.warc.gz

## Old CCF ARCs

- prefix: s3://commoncrawl/
- crawl-001/2008/06/19/0/1213886083018_0.arc.gz

## Cuil 2012

- prefix: not public
- domainshard-corpus5-large-merge-rev1.00004-of-25000.1000000sample.v1.arc.gz

## TODO

- CC-NEWS - old, pre-upgrade, post-upgrade
- ArchiveTeam WARCs (not megaWARCs)
- ArchiveIt old and new, for various flavors
- wget generates WARCs
- HTTrack used to make ARC files ?
- https://github.com/arquivo/roteiro2arc ? no examples in the repo
