Metadata-Version: 2.4
Name: ytcap
Version: 0.1.0
Summary: Extract YouTube metadata and subtitles into JSON and JSONL outputs.
Author: Özkan Öztürk
License-Expression: MIT
Project-URL: Homepage, https://github.com/ozkozturk/ytcap
Project-URL: Documentation, https://github.com/ozkozturk/ytcap#readme
Keywords: youtube,subtitles,metadata,jsonl,yt-dlp
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Text Processing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: yt-dlp>=2026.06.09
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# ytcap

`ytcap` is a Python CLI project for extracting **video metadata** and **subtitle files** from YouTube video, batch, and playlist sources, then turning them into reusable JSON and JSONL outputs.

It is designed for workflows where you have YouTube video URLs, video IDs, or playlists and want structured metadata plus subtitles for search, indexing, dataset preparation, education, or analysis.

## Project Status

This repository is in an early planning and implementation stage. The first target release is:

```text
0.1.0
```

Currently implemented:

- Importable Python package scaffold.
- Package version exposed as `ytcap.__version__`.
- Basic CLI entry point with `ytcap --help` and `ytcap --version`.
- CLI command structure for `inspect`, `video`, and `export`.
- `batch` command to process multiple video URLs or IDs from a text file, with run manifest logging, `--resume` and `--skip-existing` support.
- `yt-dlp` adapter support for `inspect` metadata extraction.
- Normalized video metadata mapping and inspect JSON summary output.
- Tested subtitle source selection for `manual`, `auto`, and `any` normalized tracks.
- Controlled subtitle format validation for `srt` and `vtt`.
- Standard output directory layout creation for `video --out`.
- `video` command metadata JSON writing and selected SRT/VTT subtitle file download.
- SRT/VTT cue parsing, cue-level JSONL writer helpers, and basic sentence-level segmentation helpers.
- `export` command conversion of existing SRT/VTT files to cue-level or sentence-level JSONL.
- `playlist` command to process videos inside a YouTube playlist with `--limit`, `--start`, and `--end` range controls, run manifest logging, `--resume`, `--skip-existing`, and `--dry-run`.
- Safe validation for dynamic output filename parts to prevent path traversal from user input or extractor metadata.

## Core Decisions

| Decision | Target |
|---|---|
| Language | Python |
| Minimum Python version | Python 3.11+ |
| CLI approach | Standard library first: `argparse` |
| Test approach | Standard library first: `unittest` |
| YouTube Data API | Not used |
| Metadata and subtitle extraction | `yt-dlp>=2026.06.09` is the core extractor dependency |
| Video/audio downloads | Out of scope for the MVP |
| Output format | JSON for metadata, JSONL for segment and sentence output |
| Distribution target | PyPI and `pipx install ytcap` |
| Additional dependencies | Require justification and approval before being added |

## Planned Capabilities

The MVP is intended to:

- Accept a single YouTube video URL or video ID.
- Save normalized video metadata as JSON.
- Find subtitles for a requested language.
- Prefer manual subtitles when available, with optional fallback to automatic subtitles.
- Save subtitles as SRT or VTT.
- Convert subtitles to cue-level or sentence-level JSONL.
- Report videos with missing or failed subtitle extraction.

Later releases may add:

- A dedicated retry command for failed records.
- PyPI publication.
- Automated test and release workflows through GitHub Actions.

## Non-Goals

`ytcap` intentionally does not:

- Use the official YouTube Data API.
- Manage API keys or OAuth flows.
- Download video or audio files as an MVP goal.
- Redistribute YouTube content or bypass access restrictions.
- Start with advanced NLP sentence segmentation libraries; simple, testable heuristics come first.

## Planned CLI Examples

These commands show the intended user experience. Some commands may not be implemented yet.
The current `inspect` command uses `yt-dlp` for metadata and subtitle
availability summaries. The `video` command extracts metadata through
`yt-dlp`, writes normalized metadata JSON, selects a matching subtitle track,
and saves the selected SRT/VTT subtitle file. SRT/VTT cue parsers,
cue-level JSONL writer helpers, and basic punctuation-based sentence
segmentation helpers are wired into the `export` command for existing
subtitle files.

### Inspect One Video

```bash
ytcap inspect --url "https://www.youtube.com/watch?v=VIDEO_ID"
```

This should answer:

- Is the video reachable?
- Can metadata be extracted?
- Which subtitle languages are available?
- Are subtitles manual, automatic, or both?

### Extract Metadata and Subtitles

```bash
ytcap video --url "https://www.youtube.com/watch?v=VIDEO_ID" --lang en --source any --format srt --out ./data
```

| Part | Meaning |
|---|---|
| `ytcap` | CLI application |
| `video` | Single-video processing command |
| `--url` | Video URL |
| `--lang en` | Request English subtitles |
| `--source any` | Try manual subtitles first, then automatic subtitles |
| `--format srt` | Save subtitles as SRT |
| `--out ./data` | Write outputs under `./data` |

Implemented source selection behavior uses exact language and format matches:
`manual` selects only manual subtitles, `auto` selects only automatic subtitles,
and `any` tries manual first before falling back to automatic subtitles.
Implemented subtitle format validation currently accepts `srt` and `vtt`; other
values return an `UNSUPPORTED_FORMAT` error before extraction work starts.
When run without `--dry-run`, `video` writes normalized metadata to
`videos/{video_id}.info.json` and the selected subtitle file to
`subtitles/{video_id}.{lang}.{source}.{format}` under the output directory.
If the requested subtitle cannot be selected or downloaded, the command returns
a controlled error without leaving a new partial metadata file behind.

### List Available Subtitles

```bash
ytcap inspect --url "https://www.youtube.com/watch?v=VIDEO_ID" --list-subs
```

### Convert Existing Subtitles to JSONL

```bash
ytcap export --input ./data/subtitles/VIDEO_ID.en.manual.srt --segments cue --out ./data/normalized
```

```bash
ytcap export --input ./data/subtitles --segments sentence --out ./data/normalized
```

The `export` command reads existing `.srt` and `.vtt` files and writes JSONL
records to `{video_id}.{lang}.{segments}.jsonl` under the output directory. It
infers `video_id`, language, and source from names such as
`VIDEO_ID.en.manual.srt`; when the source is missing, JSONL records use
`"source":"unknown"`. `--video-id` and `--lang` may override metadata for a
single file input.

Dynamic filename parts such as video ID, language, source, format, segment type,
and run ID are validated before paths are built. Empty values, path separators,
control characters, absolute paths, `.` and `..` are rejected.

### Process a Batch File

```bash
ytcap batch --input videos.txt --lang en --source any --format srt --resume --skip-existing --out ./data
```

This command parses the input file and processes each URL/ID. It creates a run
manifest under `runs/{run_id}.manifest.json` keeping track of execution
statistics, output files, and errors. Failed attempts are appended to
`failed/failed.jsonl`. `--resume` skips entries completed in the latest
manifest and retries previous failures, while `--skip-existing` skips videos
whose metadata and subtitle files already exist for the requested language,
source, and format. `--dry-run` reports the batch plan without writing files or
creating output directories.

#### Batch Input File Format

The `--input` file for the `batch` command is a plain text file containing one YouTube video URL or video ID per line.
- Empty lines and lines containing only whitespace are ignored.
- Lines starting with `#` (with optional leading whitespace) are ignored as comment lines.
- Inline comments starting with `#` are supported, and the comment text plus any preceding whitespace are ignored.

Example input file:
```text
# This is a comment line
dQw4w9WgXcQ                  # Rick Astley - Never Gonna Give You Up
https://youtu.be/jNQXAC9IVRw # Another video URL
```

### Process a Playlist

```bash
ytcap playlist --url "https://www.youtube.com/playlist?list=PLAYLIST_ID" --start 1 --limit 50 --lang en --source any --format srt --out ./data
```

The `playlist` command uses `yt-dlp` flat playlist extraction to collect video
entries without the official YouTube Data API, then processes each video with
the same metadata and subtitle flow as `video`. `--start` is 1-based, `--end`
is inclusive, and `--limit` caps the selected range. `--resume` continues only
from a matching playlist run manifest, while `--skip-existing` skips videos
only when matching metadata and subtitle files already exist.


## Output Layout

```text
data/
  videos/
    VIDEO_ID.info.json
  subtitles/
    VIDEO_ID.en.manual.srt
  normalized/
    VIDEO_ID.en.cue.jsonl
  runs/
    RUN_ID.manifest.json
  failed/
    failed.jsonl
```

Example cue-level JSONL line:

```json
{"schema_version":"0.1","type":"cue","video_id":"VIDEO_ID","language":"en","source":"manual","start":1.0,"end":3.5,"text":"Example subtitle text.","cue_index":1}
```

Sentence-level segmentation uses a simple standard-library heuristic that
splits on `.`, `?`, and `!`. Timing is marked with a strategy such as
`cue_exact`, `cue_merge`, `heuristic`, or `unknown` because sentence boundaries
can fall inside or across subtitle cues.

## Documentation

| File | Purpose |
|---|---|
| `USAGE.md` | Usage boundaries, limitations, and responsible use notes |
| `CLI_REFERENCE.md` | Planned commands, flags, behavior, and error codes |
| `OUTPUT_FORMAT.md` | Target JSON and JSONL output formats |
| `RELEASE.md` | Packaging and release process |
| `CONTRIBUTING.md` | Contributor expectations |
| `SECURITY.md` | Security policy and sensitive data rules |

## Development Install

For local development, use Python 3.11 or newer:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
```

This installs `yt-dlp>=2026.06.09` as the runtime extractor dependency. Unit
tests use fixtures and mocks instead of making real YouTube or network calls.

Smoke-test the current CLI:

```bash
ytcap --help
ytcap --version
```

## Installation Target

The long-term installation target is:

```bash
pipx install ytcap
```

Expected usage:

```bash
ytcap --help
```

```bash
ytcap video --url "https://www.youtube.com/watch?v=VIDEO_ID" --lang en --source any
```

## License

This project is planned for release under the MIT License. See `LICENSE.md` for details.
