Metadata-Version: 2.4
Name: reap-cli
Version: 0.2.0
Summary: Recursive extraction and parsing of firmware and partition images
Author: Blackbox Research
License-Expression: Apache-2.0
Keywords: firmware,forensics,reverse-engineering,android,partition,extraction,unpacker
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: System :: Archiving
Classifier: Topic :: System :: Filesystems
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: ext4>=1.2.2
Requires-Dist: lz4>=4.0.0
Requires-Dist: zstandard>=0.22.0
Provides-Extra: dev
Requires-Dist: bandit>=1.7; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: hypothesis>=6.100; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: py7zr>=0.21; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: twine>=6.1; extra == "dev"
Requires-Dist: cryptography>=43.0; extra == "dev"
Provides-Extra: signing
Requires-Dist: cryptography>=43.0; extra == "signing"
Provides-Extra: fuzz
Requires-Dist: atheris>=2.3; platform_system != "Windows" and extra == "fuzz"
Dynamic: license-file

# REAP

**Recursive Extraction And Parsing** — a general-purpose CLI tool for identifying and recursively extracting firmware and partition images. Works with raw eMMC/flash dumps, individual partition images, full-disk images (GPT or Rockchip PARM), and forensic disk images from a wide range of embedded Linux and Android devices.

Pure Python. No root, no FUSE, no mounting, no Linux kernel modules. Runs on macOS, Linux, and Windows.

## What it does

Point it at a directory of partition `.bin` files, a single image, or a set of 7z archives and it will:

1. **Identify** each image's format via magic bytes (nearly 50 format signatures)
2. **Annotate** what partition it is (boot, recovery, system, userdata, etc.) by reading ext4 superblock metadata and analyzing ramdisk contents
3. **Extract** contents recursively -- e.g. a boot image yields a kernel + ramdisk; the ramdisk decompresses to a cpio archive; the cpio extracts to a filesystem tree
4. **Analyze** kernels (version, config, build paths, kallsyms symbol table), bootloaders (U-Boot environment, embedded DTBs), and unknown partitions (forensic hex dump, strings, SHA256)
5. **Report** everything found in text, JSON, CSV, and deterministic SHA-256 forensic manifests, including FBE encryption detection

## Supported formats

### Partition tables and disk layouts

| Format | Detection | Extraction |
|--------|-----------|------------|
| GPT partition table | `EFI PART` at offset 0x200 or 0x1000 (UFS 4K sectors) | Individual partition images |
| MBR / DOS partition table | `0x55AA` at offset 510 + a valid primary entry (not a FAT VBR or GPT protective MBR) | Primary and bounded EBR logical partitions carved at their LBA-start offsets, with cycle/overlap checks, then recursed. |
| Rockchip PARM partition table | `PARM` at offset 0 | Individual partition images (RK29xx/RK3xxx flash dumps) |
| Android super.img (LP metadata) | `0x67446C70` at offset 0x1000 | Logical partition images (system, vendor, product, etc.) |

### Android boot and kernel

| Format | Detection | Extraction |
|--------|-----------|------------|
| Android Boot Image (v0--v4) | `ANDROID!` magic | Kernel, ramdisk, second-stage, recovery DTBO, DTB |
| ARM zImage | `0x016F2818` at offset 0x24 | Decompressed vmlinux, kernel config, version string, source paths, kallsyms, all strings |
| ARM64 Image | `ARM\x64` at offset 0x38 | Kernel config, version string, source paths, kallsyms, all strings |
| Raw ARM kernel binary | MSR CPSR instruction + `Linux version` string | Kernel config, version string, source paths, kallsyms, all strings |
| Device Tree Blob (DTB) | `0xD00DFEED` | Extracted DTB, optional `dtc` decompile to DTS |
| DTBO container | `0xD7B7AB1E` | Individual DT overlay entries |

### Bootloaders and firmware

| Format | Detection | Extraction |
|--------|-----------|------------|
| U-Boot uImage | `0x27051956` | Unwrapped payload (kernel, ramdisk, firmware, device tree, etc.) |
| U-Boot binary | `U-Boot <version>` string, 64 KB--4 MB | Default environment, embedded DTBs, strings |
| U-Boot environment | CRC32 + key=value pairs, power-of-2 size | Parsed environment variables |
| Samsung Exynos boot partition | BL1 header pointer + `Exynos BL` label | bl1.bin, u-boot.bin, tzsw.bin |
| Rockchip KRNL wrapper | `KRNL` at offset 0 | Unwrapped payload (re-identified as gzip, zImage, etc.) |
| ELF binary | `\x7fELF` magic | Metadata dump (class, machine, entry point), strings |
| AVB vbmeta | `AVB0` / `AVBf` | Metadata dump (version, algorithm, rollback index, flags) |

### Encrypted firmware containers

| Format | Detection | Extraction |
|--------|-----------|------------|
| IM\*H firmware container | `IM*H` at offset 0 or 0x400 | Header parse (version, module name/type, chunk table, key family). Encrypted chunks (RTOS, kernel, TZOS, DTB, etc.) extracted as raw `.bin` files. Decryption is out of scope for this tool. |
| Ambarella environment (UNR0) | `UNR0` + `0x5AA5` flags | Boot config, A/B slot status, firmware versions, bootloader logs |
| BitLocker (BDE) volume | `-FVE-FS-` OEM ID at offset 3, *or* BitLocker volume GUID at offset 0x10 of an NTFS / FAT-style boot sector | Boot-sector + FVE metadata parse: variant (Vista vs. Win 7+ / To-Go), volume GUID, encryption method (AES-128/256, CBC ± Elephant diffuser, XTS), dataset GUID, FVE block locations, full inventory of key protectors (TPM, TPM+PIN, password, recovery password, startup key, smart card, clear key, etc.). Surfaces clear-key suspended volumes prominently. Decryption is out of scope. |
| LUKS1 / LUKS2 volume | `LUKS\xBA\xBE` magic at offset 0; version word distinguishes LUKS1 from LUKS2 | LUKS1: cipher / mode / hash / key size / UUID + per-keyslot PBKDF2 iterations, salt, active flag. LUKS2: binary header + JSON metadata — segments (cipher), keyslots (PBKDF2 vs. Argon2i / Argon2id with memory + parallelism), tokens (systemd-cryptenroll TPM2 / FIDO2 / PKCS11 bindings), digests. Decryption is out of scope. |

### Filesystems

| Format | Detection | Extraction |
|--------|-----------|------------|
| ext4 | `0xEF53` at offset 0x438 | Full filesystem tree with FBE encryption detection |
| FAT12/16/32 | `0xEB`/`0xE9` + `0x55AA` at 510 | Full filesystem tree (LFN support) |
| exFAT | `EXFAT   ` OEM ID at offset 3 | Full filesystem tree (long names, fragmented + NoFatChain runs) |
| NTFS | `NTFS    ` OEM ID at offset 3 + `0x55AA` at 510 | Full filesystem tree (resident + non-resident `$DATA`, LZNT1 compression, `$ATTRIBUTE_LIST`, `$INDEX_ROOT` + `$INDEX_ALLOCATION`, ADS, USA fixup; EFS-encrypted files marked, reparse points recorded as text) |
| cramfs | `0x28cd3d45` at offset 0 + `Compressed ROMFS` at 16 | Full filesystem tree (zlib blocks, little- and big-endian) |
| romfs | `-rom1fs-` at offset 0 | Full filesystem tree (files, dirs, symlinks, hard links) |
| squashfs | `hsqs` (little-endian) / `sqsh` (big-endian) at offset 0 | Full filesystem tree for v4.0 little-endian images — inode/directory metadata + data/fragment blocks. Compressors: gzip, xz/lzma, lz4, zstd (if `zstandard` installed); lzo and big-endian images are flagged, not decoded |
| JFFS2 | node magic `0x1985` (`85 19` LE / `19 85` BE) at offset 0 | Full filesystem tree (log-structured node scan, newest-version resolution; none/zero/copy/zlib/rtime compressors; lzo/rubin flagged, not decoded) |
| UBI | `UBI#` erase-counter header at offset 0 (per PEB) | Reassembles each logical volume from its PEBs (infers PEB size, orders LEBs, parses volume names) and carves each volume as a child image for recursion |
| UBIFS | node magic `0x06101831` at offset 0 | CRC-validated linear node-scan recovery (tolerates mid-commit/partial dumps where the committed index is inconsistent): rebuilds the tree from surviving newest-`sqnum` inode/dentry/data nodes, decompresses none/zlib (lzo/zstd if the optional lib is present), zero-fills + flags missing blocks, and puts unreferenced inodes under `__ORPHANS__/` |
| EROFS | `0xE0F5E1E2` at offset 0x400 | Flat plain/inline files plus conservative LZ4 full-index and 4-byte compact-index, one-pcluster mappings; advanced/chunk/fragment/2-byte layouts fail closed with typed warnings. |
| F2FS | `0xF2F52010` at offset 0x400 | CRC-validated checkpoint-pack recovery, checkpoint-authoritative NAT selection, direct/indirect/double-indirect nodes, sparse files, inline data and inline directories. Encrypted files, compressed clusters, and special inodes fail closed. |

### Compression and archives

| Format | Detection | Extraction |
|--------|-----------|------------|
| gzip | `1F 8B` | Decompressed content |
| LZ4 frame | `04 22 4D 18` | Decompressed content |
| LZ4 legacy | `02 21 4C 18` | Decompressed content (Android ramdisk format) |
| LZMA | `5D 00 00` | Decompressed content |
| bzip2 | `BZh` | Decompressed content |
| XZ | `FD 37 7A 58 5A 00` | Decompressed content |
| cpio newc | `070701` / `070702` | Files, directories, symlinks (as text files with `-> target`) |
| 7z archive | `37 7A BC AF 27 1C` | Full decompression (supports split `.7z.001` parts) |
| ZIP / TAR / Odin `.tar.md5` | Native signatures and validated TAR trailer | Safely extracts regular members; rejects traversal, links/devices, collisions, corruption, excessive members and expansion; Odin MD5 is verified. |
| Zstandard | `28 B5 2F FD` | Bounded streamed decompression |
| Android sparse image | `0xED26FF3A` | Converted to raw image, then re-identified and extracted |

### Device-specific partitions

| Format | Detection | Extraction |
|--------|-----------|------------|
| Android devinfo | `ANDROID-BOOT!` magic | Lock status, tamper flags |
| ArduPilot / Autel DataFlash flight log | `0xA3 0x95` head + FMT type `0x80` | Self-describing FMT schema, per-message CSVs, decoded firmware versions, and a GPS track (GPX + KML). Auto-detects the Autel 8-byte FMT name field vs. stock ArduPilot's 4-byte field. |
| Qualcomm modemst (EFS) | `IMGEFS` marker in first 64 bytes | Forensic scan (SHA256, strings, hex dump) |
| BMP image | `BM` + valid DIB header | Trimmed BMP (strips partition padding) |
| Boot logo container | ASCII count/sizes header + BMP at 0x200 | Individual BMP images |
| Empty / zeroed | All-zero content | Verified-empty marker with likely purpose annotation |

## Installation

Requires **Python 3.10+** (supported through Python 3.13).

From PyPI:

```bash
pip install reap-cli
```

> The PyPI distribution is `reap-cli` because the bare `reap` name on PyPI is held by an unrelated, long-abandoned 2012 package. We are pursuing a [PEP 541](https://peps.python.org/pep-0541/) transfer. The installed CLI command is `reap` regardless.

From source (for development):

```bash
git clone https://gitlab.com/blackbox-research/reap
cd reap
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

Dependencies (installed automatically):
- `ext4` -- pure-Python ext4 filesystem reader (no FUSE/mounting)
- `lz4` -- LZ4 decompression for Android ramdisks
- `zstandard` -- Zstandard archives and filesystem blocks

Third-party plugins that add format handlers or detectors are discovered automatically via the `reap.plugins` entry point group. See the architecture section below.

Inspect installed plugin metadata without importing plugin code using
`reap plugins`; add `--load` to explicitly validate registrations. Extraction
supports repeatable `--enable-plugin NAME` and `--disable-plugin NAME` options.
Selections and distribution versions are bound into resume snapshots, and
incompatible, conflicting, or unknown selections fail closed with exit 7.
Plugin API v1 runs trusted Python code in-process; its isolation limitation and
metadata contract are documented in [docs/API.md](docs/API.md).
Inventory is metadata-only, but `--load` and extraction import and execute the
selected plugin with the same filesystem, network, and process permissions as
REAP. Only enable plugins whose package and publisher you trust.

## Usage

```
reap <input_path> [options]
```

`input_path` can be a single image file, a directory containing partition images, or a set of 7z archives.

### Options

| Flag | Description |
|------|-------------|
| `-o DIR` | Exact output directory for a single file; output root for a directory (defaults: `<file>_unpacked/` or `<directory>/unpacked/`) |
| `--identify-only` | Print format identification only, no extraction |
| `--skip-ext4` | Skip ext4 filesystem extraction (useful for huge partitions) |
| `--skip-archives` | Skip 7z archive extraction |
| `--force-archives` | Force archive extraction even when `physicalImage/` already exists |
| `--no-recursive` | Don't recurse into extracted children |
| `--salvage`, `--carve-scan` | For images with an unparseable partition table (identified as `Unknown`), scan for embedded ext4/FAT/exFAT/NTFS filesystems and a UBI area (`UBI#` after a boot region) and carve them out for extraction |
| `--max-depth N` | Maximum recursion depth (default: 10) |
| `--max-items N` | Total extraction-item budget (default: 100000; 0 = unlimited) |
| `--max-output-bytes N` | Per-artifact extraction cap (default: max(1 GiB, 200× input); 0 = unlimited) |
| `--max-total-output-bytes N` | Thread-safe whole-run output cap (default: 20 GiB; 0 = unlimited) |
| `--timeout SECONDS` | Cooperative deadline; timeout exits 124 and cleans partial files |
| `--progress none\|human\|json` | Progress on stderr; JSON mode is JSON Lines |
| `--resume` | Resume only from a matching hash-validated journal; stale/missing state exits 3 |
| `-y`, `--assume-yes` | Use the default archive-skip answer without prompting |
| `-j, --jobs N` | Parallel extraction workers (0=auto, 1=sequential; default: auto) |
| `-v` | Verbose output (INFO level) |
| `-vv` | Debug output |
| `--report text\|json\|csv\|both\|all` | Report format (`all` emits all three; default: both) |
| `--manifest none\|json\|csv\|both` | Portable SHA-256 evidence manifest (default: both) |

Detached authenticity is opt-in and never creates or discovers keys:

```console
pip install 'reap-cli[signing]'
chmod 600 investigator-ed25519-private.pem
reap sign-manifest output/manifest.json --private-key investigator-ed25519-private.pem --signature output/manifest.sig.json
reap verify-signature output/manifest.json --signature output/manifest.sig.json --public-key trusted-investigator-ed25519.pem
```

Private keys must be unencrypted Ed25519 PEM files with owner-only permissions
on POSIX systems. Keep them outside evidence/output trees. Public trust is
always an explicit `--public-key` path; key generation, persistence, lookup,
and logging are deliberately out of scope. Signature mismatch exits 6;
malformed inputs, unsafe key permissions, missing signing support, and I/O
errors exit 5. Existing `reap verify` content verification and exit codes are
unchanged and can be run before or after authenticity verification.

### Examples

**Identify all partitions in a dump:**
```bash
reap ./physicalImage --identify-only
```

**Full extraction (skip large ext4 partitions):**
```bash
reap ./physicalImage --skip-ext4 -v
```

**Extract a single boot image:**
```bash
reap boot.img -o ./boot_extracted -v
```

**Extract a directory of 7z archives (split parts supported):**
```bash
reap ./archives/ -v
```

**Parallel extraction with 4 workers:**
```bash
reap ./physicalImage -j 4 -v
```

### Output structure

For a boot image, the recursive extraction produces:

```
boot_unpacked/
    kernel_info.txt          # Kernel analysis summary
    kernel_config.txt        # Build-time .config (if IKCONFIG enabled)
    kernel_source_paths.txt  # Build-time source paths
    kernel_strings.txt       # All embedded ASCII strings
    kallsyms.txt             # Kernel symbol table (if present)
    vmlinux                  # Decompressed kernel binary
    ramdisk_unpacked/
        init                 # cpio filesystem tree
        init.rc
        fstab.*
        sbin/
        ...
```

For a directory of partitions, you get a subdirectory per partition plus reports:

```
physicalImage/unpacked/
    report.txt               # Human-readable report
    report.json              # Machine-readable report
    report.csv               # Flat machine-readable report (with --report csv/all)
    manifest.json            # Versioned portable evidence manifest
    manifest.csv             # Flat evidence manifest
    mmcblk0p1_unpacked/      # boot image contents
    mmcblk0p2_unpacked/      # DTB contents
    mmcblk0p3_unpacked/      # recovery image contents
    mmcblk0p4_unpacked/      # system filesystem tree
    ...
```

### Partition annotation

The tool automatically identifies partition roles by:
- Reading the ext4 superblock `s_last_mounted` field (e.g. `/system`, `/data`, `/cache`)
- Analyzing boot image ramdisks for `/sbin/recovery` to distinguish boot vs recovery
- Parsing U-Boot uImage type fields (kernel, ramdisk, firmware, device tree)
- Parsing IM\*H firmware module names and types (bootloader, kernel, RTOS)
- Recognizing format-specific roles (DTB, vbmeta, DTBO, sparse, super, modemst)
- Inferring empty partition purpose from size (<=4 MB zeroed = likely misc or metadata)

Annotations appear in reports and verbose output as labels like `(recovery)`, `(system)`, `(userdata)`, etc.

### FBE encryption detection

When extracting ext4 filesystems with File-Based Encryption (FBE), the tool:
- Detects the encryption superblock flag and per-inode encryption flags
- Hex-encodes encrypted filenames for safe extraction
- Writes `encrypted_paths.txt` listing all encrypted files and directories
- Reports encryption algorithms (AES-256-XTS, AES-256-GCM, etc.) in JSON output

## Architecture

```
reap/
    cli.py              # Argument parsing, entry point
    identify.py         # Ordered detection registry (52 core formats)
    annotate.py         # Partition role inference
    pipeline.py         # Recursive extraction orchestrator (parallel workers)
    report.py           # Atomic text + JSON + CSV reports (FBE-aware)
    manifest.py         # Deterministic portable JSON + CSV evidence manifests
    limits.py           # Output budgets, bounded writers, cancellation
    resume.py           # Atomic hash-validated resume journals
    handlers/
        __init__.py     # BaseHandler ABC, handler registry
        ambarella_env.py # Ambarella UNR0 boot environment
        avb.py          # AVB vbmeta metadata
        bitlocker.py    # BitLocker (BDE) detection + FVE metadata + protectors
        bmp.py          # BMP image (partition padding trim)
        boot_img.py     # Android boot image (v0--v4)
        bootlogo.py     # Boot logo container (multiple BMPs)
        compression.py  # gzip, LZ4, LZMA, bzip2, XZ
        cpio_handler.py # cpio newc archives
        dataflash.py    # ArduPilot/Autel DataFlash flight log (schema, CSVs, GPS track)
        devinfo.py      # Android devinfo (lock status)
        dji_imah.py     # IM*H encrypted firmware container (header parse, encrypted chunks)
        dtb.py          # Device Tree Blob
        dtbo.py         # DTBO container
        elf.py          # ELF binary metadata + strings
        ext4_handler.py # ext4 filesystem (FBE detection, dir_index fallback)
        exynos_boot.py  # Samsung Exynos eMMC boot partition
        mbr.py          # MBR/DOS partition table (splits primary partitions)
        fat.py          # FAT12/16/32 filesystem
        gpt.py          # GPT partition table (512-byte + 4K UFS sectors)
        luks.py         # LUKS1 + LUKS2 detection + keyslot / token metadata
        modemst.py      # Qualcomm modem EFS partition
        ntfs.py         # NTFS filesystem (MFT + runlists + LZNT1 + indexes)
        raw.py          # Empty + unknown fallback (forensic scan)
        salvage.py      # --salvage carve-by-scan for unparseable partition tables
        raw_kernel.py   # Raw ARM kernel binary
        rk_krnl.py      # Rockchip KRNL wrapper
        rkparm.py       # Rockchip PARM partition table
        seven_zip.py    # 7z archive (split-part support)
        sparse_img.py   # Android sparse -> raw conversion
        super_img.py    # super.img LP metadata
        uboot_bin.py    # U-Boot binary (environment, embedded DTBs)
        uboot_env.py    # U-Boot environment block
        uimage.py       # U-Boot uImage wrapper
        zimage.py       # ARM zImage / ARM64 Image kernel extraction
        _kernel_utils.py # Shared kernel analysis (version, config, kallsyms)
```

Each handler implements `BaseHandler.extract()` and returns an `ExtractionResult` with optional children for recursive processing. Versioned public interfaces, plugin keys and conflict rules are documented in [docs/API.md](docs/API.md).

The pipeline orchestrator (`pipeline.py`) drives identify → annotate → dispatch → recurse. Children can run concurrently while sharing one reservation-based output ledger and cancellation token. See [architecture](docs/ARCHITECTURE.md), [schemas](docs/SCHEMAS.md), and [limitations](docs/LIMITATIONS.md).

## Symlink handling

Symlinks found inside ext4 filesystems and cpio archives are **not** created as OS symlinks (which can cause issues on some platforms and create security risks with path traversal). Instead, they're written as small text files containing `-> target` and recorded in the extraction metadata / JSON report.

## Running tests

```bash
pip install -e ".[dev]"
pytest tests/ -v
```

The comprehensive synthetic-data suite covers format detection, handler extraction, hostile-input bounds, kernel analysis (kallsyms), pipeline orchestration, and forensic scanning; no real image files are required.

## Known limitations

- **EROFS and F2FS**: Useful conservative subsets are extracted. Advanced layouts intentionally fail closed; see [the exact support matrix](docs/LIMITATIONS.md).
- **Encrypted partitions**: FBE-encrypted ext4 partitions are detected and documented, but file contents remain encrypted. The tool does not perform Android FDE/FBE decryption. EFS-encrypted files on NTFS volumes are similarly skipped, with a per-file marker recording the encryption. BitLocker and LUKS volumes are detected and their full encryption-scheme metadata is surfaced, but the encrypted payload itself is not decrypted — pipe the source image through `dislocker` / `libbde` (BitLocker) or `cryptsetup` (LUKS) for that.
- **IM\*H decryption**: Out of scope. The core tool parses IM\*H headers and extracts encrypted chunks as raw `.bin` files. Producing plaintext requires AES keys that are not distributed with this tool.
- **Large partitions**: Extracting a 54 GB ext4 partition takes time and disk space. Use `--skip-ext4` to skip these, or extract individual partitions as needed.
- **7z extraction**: Requires system `7z` binary for split archives; falls back to `py7zr` for single files.
- **Symlinks**: Recorded as text files, not created as actual OS symlinks.
- **Text files**: Plain-text metadata files (.txt, .sha256, .xml, README) in the input directory are detected and skipped rather than subjected to forensic extraction.
