Metadata-Version: 2.4
Name: chunkops
Version: 0.1.1
Summary: CLI tool for detecting duplicate content in PDF documents
Home-page: https://chunkops.ai
Author: ChunkOps
Author-email: ChunkOps <hello@chunkops.ai>
License-Expression: MIT
Project-URL: Homepage, https://chunkops.ai
Project-URL: Documentation, https://chunkops.ai/docs
Project-URL: Repository, https://github.com/chunkops/chunkops-cli
Keywords: rag,pdf,duplicate-detection,document-analysis
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyMuPDF>=1.23.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: langchain-text-splitters>=0.2.0
Requires-Dist: rich>=13.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# ChunkOps CLI

A command-line tool for detecting duplicate and near-duplicate content in PDF documents.

## Installation

### From PyPI (when published)
```bash
pip install chunkops
```

### From Source (Development)
```bash
cd chunkops-cli
python3 -m pip install -e .
```

**Note:** Use `python3 -m pip` if `pip` command is not found in your PATH.

**Important:** Make sure your virtual environment is activated:
```bash
source /path/to/venv/bin/activate  # Activate your venv
chunkops --version                  # Should work now
```

Or use the full path:
```bash
/path/to/venv/bin/chunkops --version
```

## Usage

Scan a folder of PDFs and generate a JSON report of duplicates:

```bash
chunkops scan /path/to/pdf/folder --output report.json
```

### Options

- `--output`, `-o`: Path to output JSON file (default: `duplicates_report.json`)
- `--threshold`: Similarity threshold for near-duplicates (0.0-1.0, default: 0.90)
- `--exact-threshold`: Threshold for exact duplicates (default: 1.0)
- `--verbose`, `-v`: Enable verbose output

### Example

```bash
chunkops scan ./documents --output duplicates.json --threshold 0.95
```

## Output Format

The JSON report contains:

```json
{
  "scan_date": "2024-01-15T10:30:00Z",
  "total_files": 10,
  "total_chunks": 245,
  "duplicates": [
    {
      "type": "EXACT_DUPLICATE",
      "file_a": "document1.pdf",
      "file_b": "document2.pdf",
      "similarity": 1.0,
      "chunk_a_preview": "...",
      "chunk_b_preview": "..."
    },
    {
      "type": "NEAR_DUPLICATE",
      "file_a": "document3.pdf",
      "file_b": "document4.pdf",
      "similarity": 0.95,
      "chunk_a_preview": "...",
      "chunk_b_preview": "..."
    }
  ],
  "summary": {
    "exact_duplicates": 2,
    "near_duplicates": 5,
    "unique_content": 238
  }
}
```

## Limitations

- **Local only**: No cloud sync, runs entirely on your machine
- **No UI**: Command-line interface only
- **PDF only**: Currently supports PDF files only
- **Single scan**: Each scan is independent (no persistent history)

## License

MIT

## Support

Visit [chunkops.ai](https://chunkops.ai) for more information.

