Metadata-Version: 2.4
Name: netintel-ocr
Version: 0.1.5
Summary: A tool for extracting network intelligence from PDFs using OCR and AI.
Project-URL: Homepage, https://github.com/NetVisionML/NetIntelOCR
Project-URL: Issues, https://github.com/NetVisionML/NetIntelOCR/issues
Project-URL: Changelog, https://github.com/NetVisionML/NetIntelOCR/changelog.md
Requires-Python: >=3.10
Requires-Dist: opencv-python-headless>=4.5.0
Requires-Dist: pillow>=11.2.1
Requires-Dist: pymupdf>=1.26.1
Requires-Dist: requests>=2.32.4
Requires-Dist: tqdm>=4.67.1
Description-Content-Type: text/markdown

# NetIntel-OCR (Network Intelligence OCR)

🔍 **Extract Network Intelligence from PDFs with AI-Powered OCR**

NetIntel-OCR is a specialized tool designed for network engineers and security professionals to automatically extract, analyze, and convert network diagrams from PDF documents into structured formats. Using advanced OCR and AI technologies, it identifies network topologies, components, and connections, transforming them into actionable intelligence.

## 🎯 Key Capabilities

### Network Intelligence Extraction
- **Automatic Network Detection**: AI-powered identification of network diagrams in documents
- **Component Recognition**: Identifies routers, switches, firewalls, servers, and other network elements
- **Connection Mapping**: Traces and documents network paths and relationships
- **Security Architecture Analysis**: Extracts security zones, DMZs, and trust boundaries

## ✨ Features

- 🎯 **Intelligent Hybrid Processing**: Automatically detects and processes network diagrams as Mermaid.js, text as markdown
- 📄 **PDF to Text Conversion**: Convert PDFs to markdown files locally, no token costs
- 🤖 **Multi-Model Support (v0.1.4)**: Use different models for text and network processing for optimal performance
- 🖼️ **Visual Understanding**: Turn images and diagrams into detailed text descriptions
- 🔌 **Automatic Network Detection**: No flags needed - network diagrams are detected and converted automatically
- 🎨 **Icons by Default**: Font Awesome icons automatically added to network diagrams for better visualization
- ⏱️ **Smart Timeouts**: Operations timeout gracefully with fallback to simpler methods
- 📊 **Diagram Types Supported**: Network topology, architecture diagrams, data flow diagrams, security diagrams
- 📁 **MD5-Based Organization (v0.1.4)**: Each document stored in unique folder using MD5 checksum
- 📝 **Document Index (v0.1.4)**: Automatic index.md tracking all processed documents
- 📈 **Enhanced Metrics (v0.1.4)**: Comprehensive footer with processing details, errors, and configuration
- ⚡ **Optimized Processing**: Processes up to 100 pages per run with detailed progress tracking
- 🔧 **Flexible Output**: Unified markdown format with seamlessly embedded Mermaid diagrams

## 💼 Use Cases

### Network Documentation
- Convert legacy network diagrams to modern formats
- Extract network topology from vendor documentation
- Audit and inventory network architectures

### Security Analysis
- Map security architecture from compliance documents
- Extract firewall rules and network segmentation
- Document data flow and trust boundaries

### Infrastructure Planning
- Analyze existing network designs
- Extract capacity and redundancy information
- Document interconnections and dependencies

## 📦 Requirements

- Python 3.10+
- Ollama installed and running locally or on a remote server

### Installing Ollama and the Default Model

1. Install [Ollama](https://ollama.com/)
2. Pull the default model:
```bash
ollama run nanonets-ocr-s:latest
```

### Using a Remote Ollama Server

By default, netintel-ocr connects to Ollama running on localhost. To use a remote Ollama server, set the `OLLAMA_HOST` environment variable:

```bash
# Connect to a remote Ollama server
export OLLAMA_HOST="http://192.168.1.100:11434"
netintel-ocr document.pdf

# Or run with the environment variable inline
OLLAMA_HOST="http://remote-server:11434" netintel-ocr document.pdf
```

## Installation

### From PyPI

Install the published version using pip:
```bash
pip install netintel-ocr
```

or uv:
```bash
uv tool install netintel-ocr
```

## Usage

### Default Behavior (NEW!)
By default, netintel-ocr automatically detects network diagrams and processes them as Mermaid diagrams:
```bash
# Automatic hybrid mode - detects and converts network diagrams
netintel-ocr path/to/your/file.pdf
```

### Text-Only Mode
For faster processing when you know the document contains only text:
```bash
netintel-ocr document.pdf --text-only
```

### Multi-Model Processing (NEW v0.1.4!)

Use different Ollama models optimized for specific tasks:
```bash
# Use fast OCR model for text, powerful model for diagrams
netintel-ocr document.pdf --model nanonets-ocr-s --network-model qwen2.5vl

# Fast processing with lightweight models
netintel-ocr document.pdf --model moondream --network-model bakllava

# Heavy processing for complex network diagrams
netintel-ocr document.pdf --network-model cogvlm --timeout 120
```

**Multi-Model Benefits:**
- 30-50% faster text extraction with OCR-optimized models
- Better diagram understanding with vision-language models
- Resource efficiency by using appropriate model sizes
- Flexibility to experiment with different combinations

**Recommended Model Combinations:**
| Purpose | Text Model | Network Model | Speed |
|---------|------------|---------------|-------|
| Balanced (Default) | nanonets-ocr-s | qwen2.5vl | Medium |
| Fast Processing | moondream | bakllava | Fast |
| Maximum Accuracy | qwen2.5vl | cogvlm | Slow |
| Resource Limited | moondream | llava-phi3 | Fast |

### Performance Optimization

For faster processing of network diagrams, use the `--fast-extraction` flag:
```bash
# Fast extraction mode - reduces extraction time by 50-70%
netintel-ocr document.pdf --fast-extraction

# Combine with multi-model and timeout for best performance
netintel-ocr document.pdf --model nanonets-ocr-s --network-model bakllava --fast-extraction --timeout 30
```

**Fast extraction benefits:**
- Detection: ~15 seconds (vs 30-60s standard)
- Extraction: ~20 seconds (vs 30-60s standard)
- Uses simplified prompts for quicker LLM responses
- Automatic fallback if fast extraction fails

### Command Line Options

#### Basic Options
- `--output`, `-o`: Base output directory (default: "output", documents stored in `output/<md5_checksum>/`)
- `--model`, `-m`: Ollama model for text extraction (default: "nanonets-ocr-s:latest")
- `--network-model`: Separate model for network diagram processing (NEW v0.1.4)
- `--keep-images`, `-k`: Keep the intermediate image files (default: False)
- `--width`, `-w`: Width to resize images to, 0 to skip resizing (default: 0)
- `--start`, `-s`: Start page number (default: 0, processes from beginning)
- `--end`, `-e`: End page number (default: 0, processes to end)
- `--resume`: Resume processing from checkpoint if available (NEW v0.1.5)

#### Processing Mode Options
- `--text-only`, `-t`: Skip network diagram detection for faster text-only processing
- `--network-only`: Process only network diagrams, skip regular text pages

#### Network Diagram Options (applies to default mode)
- `--confidence`, `-c`: Minimum confidence threshold for network diagram detection (0.0-1.0, default: 0.7)
- `--no-icons`: Disable Font Awesome icons in Mermaid diagrams (icons are enabled by default)
- `--diagram-only`: Only extract network diagrams without page text (by default, both are extracted)
- `--timeout`: Timeout in seconds for each LLM operation (default: 60s, increase for complex diagrams)

### Examples

#### Basic Usage (with automatic network detection)
```bash
# DEFAULT: Automatic network diagram detection (with icons)
netintel-ocr document.pdf

# Process with custom settings
netintel-ocr document.pdf --confidence 0.8

# Increase timeout for complex diagrams
netintel-ocr document.pdf --timeout 120

# Text-only mode (faster, no detection)
netintel-ocr document.pdf --text-only

# Process specific pages
netintel-ocr document.pdf --start 1 --end 5

# Use a different Ollama model
netintel-ocr document.pdf --model qwen2.5vl:latest
```

#### Specialized Processing
```bash
# Process ONLY network diagrams (skip text pages)
netintel-ocr network-architecture.pdf --network-only

# Higher confidence threshold (stricter detection)
netintel-ocr document.pdf --confidence 0.9

# Disable icons if not needed
netintel-ocr document.pdf --no-icons

# Extract only diagrams without text (faster)
netintel-ocr document.pdf --diagram-only

# Faster text-only processing
netintel-ocr text-document.pdf --text-only
```

Process large documents in sections (max 100 pages per run):
```bash
# Process first 100 pages
netintel-ocr large-document.pdf --start 1 --end 100

# Process next section
netintel-ocr large-document.pdf --start 101 --end 200

# Process specific chapter (e.g., pages 50-100)
netintel-ocr large-document.pdf --start 50 --end 100
```

## Checkpoint/Resume Capability (NEW v0.1.5)

The tool now supports automatic checkpoint saving and resume functionality for long documents:

### How It Works
- **Automatic Saving**: Processing state is saved after each page
- **Checkpoint Location**: Stored in `output/<md5>/.checkpoint/`
- **Resume on Interruption**: Use `--resume` to continue from where you left off
- **Page-Level Tracking**: Each page is tracked individually
- **Smart Skip**: Already processed pages are skipped when resuming

### Usage Examples
```bash
# Start processing a large document
netintel-ocr large-document.pdf

# If interrupted (Ctrl+C, power failure, etc.), resume processing
netintel-ocr large-document.pdf --resume

# Resume with different settings (completed pages are kept)
netintel-ocr large-document.pdf --resume --timeout 120 --network-model qwen2.5vl
```

### Resume Information
When resuming, you'll see a summary like:
```
╔════════════════════════════════════════════════════════════╗
║                  RESUME CHECKPOINT FOUND                   ║
╠════════════════════════════════════════════════════════════╣
║ Previous Processing:                                        ║
║   • Pages completed: 45/100                                ║
║   • Network diagrams found: 5                              ║
║   • Regular pages: 40                                      ║
║   • Failed pages: 0                                        ║
║                                                            ║
║ Resume Information:                                        ║
║   • Will skip 45 already processed pages                   ║
║   • Will process 55 remaining pages                        ║
║   • Starting from page 46                                  ║
╚════════════════════════════════════════════════════════════╝
```

### Benefits
- **No Lost Work**: Never lose progress on long documents
- **Resource Efficient**: Don't reprocess completed pages
- **Flexible**: Change settings when resuming
- **Automatic**: No manual intervention needed

## Processing Guidelines

### Document Size Recommendations

| Document Size | Processing Strategy | Example |
|--------------|-------------------|---------|
| 1-50 pages | Single run | `netintel-ocr doc.pdf` |
| 51-100 pages | Single run or split | `netintel-ocr doc.pdf` |
| 101-300 pages | Process in 100-page sections | See examples below |
| 300+ pages | Process key sections only | Use specific page ranges |

### Processing Large Documents

For a 250-page document:
```bash
# Section 1: Pages 1-100
netintel-ocr document.pdf --start 1 --end 100 -o output_section1

# Section 2: Pages 101-200
netintel-ocr document.pdf --start 101 --end 200 -o output_section2

# Section 3: Pages 201-250
netintel-ocr document.pdf --start 201 --end 250 -o output_section3
```

## Network Diagram Detection (Now Default!)

**NEW**: Network diagram detection is now enabled by default! No flags needed.

netintel-ocr automatically (in order):

1. **Transcribes** text content FIRST (guaranteed capture)
2. **Detects** network diagrams in PDF pages
3. **Identifies** components (routers, switches, firewalls, servers, databases, etc.)
4. **Extracts** connections and relationships
5. **Converts** to Mermaid.js format
6. **Combines** BOTH the diagram AND the page's text content
7. **Embeds** everything in unified markdown output

### Supported Network Components
- 🔀 Routers and Switches
- 🛡️ Firewalls
- 🖥️ Servers and Workstations
- 💾 Databases
- ⚖️ Load Balancers
- ☁️ Cloud Services
- 📡 Wireless Access Points

### Output Format

Network diagrams are saved as markdown with embedded Mermaid code:

```markdown
# Page 5 - Network Diagram

**Type**: topology
**Detection Confidence**: 0.95
**Components**: 8 detected
**Connections**: 12 detected

## Diagram

```mermaid
graph TB
    Router([Main Router])
    Switch[Core Switch]
    FW{{Firewall}}
    Server1[(Web Server)]
    
    Router --> FW
    FW --> Switch
    Switch --> Server1
```

## Page Text Content

This section describes the SD-WAN architecture with multiple branch offices
connecting to headquarters through various transport methods including MPLS,
broadband, and LTE connections. The solution provides path selection,
application-aware routing, and centralized management...
```

## Output Structure (Enhanced v0.1.4)

All output is organized using MD5 checksums for unique document identification:

```
output/                                    # Base directory (configurable with --output)
├── index.md                              # Master index tracking all processed documents
├── 6c928950e6b73fffe316e0ad6bba3a67/    # MD5 checksum as folder name
│   ├── markdown/                         # All transcribed content
│   │   ├── page_001.md                  # Individual page (text or diagram)
│   │   ├── page_002.md    
│   │   └── document.md                  # Complete merged document with footer metrics
│   ├── images/                          # Original page images (if --keep-images)
│   └── summary.md                       # Processing summary and statistics
└── 0611ca05dab284e943e3b00d3993d424/    # Another document's folder
    └── ...

Benefits:
- Same document won't be processed twice (deduplication)
- Easy to find previous processing results
- index.md provides overview of all processed documents
```

### Index File (output/index.md)
Automatically tracks all processed documents:
```markdown
| Filename | Timestamp | MD5 Checksum | Folder | Processing Time |
|----------|-----------|--------------|--------|----------------|
| network.pdf | 2025-08-20 14:30:15 | `6c9289...` | [📁 6c9289...](./6c9289.../) | 2m 30s |
| manual.pdf | 2025-08-20 14:35:22 | `0611ca...` | [📁 0611ca...](./0611ca.../) | 1m 45s |
```

### Enhanced Footer Metrics (NEW v0.1.4)
Every merged document includes comprehensive processing metrics:
- **Document Info**: Source file, size, MD5 checksum, pages processed
- **Processing Details**: Date/time, models used, processing time, mode
- **Quality Report**: Errors, warnings, success metrics
- **Configuration**: Settings used during processing

## Processing Modes

### Default: Hybrid Mode (Text-First)
- **Text-First Approach**: ALWAYS transcribes text before attempting diagram detection
- **Guaranteed Content**: Text is captured even if diagram processing fails
- **Automatic Detection**: Every page is analyzed for network diagrams
- **Dual Content Extraction**: Pages with diagrams include BOTH Mermaid diagram AND text content
- **Intelligent Processing**: Network diagrams → Mermaid (with icons), Text → Markdown
- **Progress Tracking**: Detailed step-by-step progress messages
- **Smart Timeouts**: Operations timeout after 60s with automatic fallback
- **Processing Time**: 30-60 seconds per page
- **Best For**: Most documents (mixed content)

### Text-Only Mode (`--text-only`)
- **No Detection**: Skip diagram detection for speed
- **Processing Time**: 15-30 seconds per page
- **Best For**: Documents with only text

### Network-Only Mode (`--network-only`)
- **Diagram Focus**: Process only network diagrams
- **Processing Time**: 30-60 seconds per diagram
- **Best For**: Network architecture documents

## Performance & Troubleshooting

### If Processing is Slow or Stuck

The tool now includes detailed progress messages showing what's happening and which models are being used:
```
  Page 3: Processing...
    Transcribing page text (nanonets-ocr-s)... Done (12.3s)  <-- Text captured first!
    Checking for network diagram (qwen2.5vl)... Done (2.1s)
    Network diagram detected (confidence: 0.90)
    Type: topology
    Extracting components (qwen2.5vl)... Done (5.1s)
    Generating Mermaid diagram (qwen2.5vl)... Done (8.2s)
    Validating Mermaid syntax... Valid (0.1s)
    Writing to file... Done (0.1s)
    Total processing time: 27.9s
```

**Important**: Text is ALWAYS transcribed first, so even if diagram processing times out or fails, you'll still have the page content.

If an operation takes too long:
- **Default timeout**: 60 seconds per operation
- **Adjust timeout**: Use `--timeout 120` for complex diagrams
- **Automatic fallback**: If LLM times out, falls back to simpler methods

### Common Issues and Fixes

#### Mermaid Syntax Errors (Robust Auto-Fix)
The tool uses a comprehensive validator to automatically fix Mermaid syntax issues:

**Phase 1 - Basic Cleanup:**
- C-style comments (`//`) → Removed or converted to Mermaid comments (`%%`)
- Curly braces in graph declarations → Removed
- Invalid syntax elements → Cleaned

**Phase 2 - Node ID Fixing:**
- Spaces in node IDs → Converted to underscores (e.g., `Data Center` → `Data_Center`)
- Special characters → Replaced with safe alternatives
- Duplicate node IDs → Automatically numbered (e.g., `Server`, `Server2`, `Server3`)

**Phase 3 - Connection Fixing:**
- Updates all connections to use fixed node IDs
- Preserves connection types and labels
- Maintains directional flow

**Phase 4 - Style Application:**
- Fixes class applications to use corrected node IDs
- Preserves styling and visual attributes

**Examples of Auto-Fixes:**
- `subgraph_DMZ` → `subgraph DMZ`
- `Data Center (HQ)` → `Data_Center_HQ` (as node ID)
- Parentheses in labels → Automatically quoted
- Multiple `Secure SD-WAN` nodes → `Secure_SD_WAN`, `Secure_SD_WAN2`, etc.

## Recent Improvements

### Version 0.1.5 (Latest - 2025-08-20)
- ✅ **Checkpoint/Resume**: Automatic saving and resume capability for long documents
- ✅ **Page-Level Tracking**: Individual page checkpoint tracking
- ✅ **Resume Summary**: Clear display of resume status and remaining work
- ✅ **Atomic Saves**: Checkpoint integrity with atomic file operations
- ✅ **Automatic Cleanup**: Checkpoints removed after successful completion

### Version 0.1.4 (2025-08-20)
- ✅ **Multi-Model Support**: Use different models for text and network processing
- ✅ **MD5-Based Output**: Unique folders per document using MD5 checksums
- ✅ **Document Index**: Automatic index.md tracking all processed documents
- ✅ **Enhanced Footer**: Comprehensive metrics in merged documents
- ✅ **Simplified Defaults**: Output to `output/` instead of timestamped folders
- ✅ **Model Progress Display**: Shows which model is being used for each operation
- ✅ **Deduplication**: Same document uses same output folder

### Version 0.1.3
- ✅ **Hybrid Mode by Default**: Automatic network diagram detection
- ✅ **Text-First Processing**: Guarantees content capture before diagram extraction
- ✅ **Fast Extraction Mode**: 50-70% faster processing option
- ✅ **Enhanced Error Recovery**: Graceful fallbacks and timeout management

### Version 0.1.0 
- ✅ **Initial pypi.org Release**
- ✅ **Fixed Mermaid syntax issues**: Automatically handles parentheses in node labels
- ✅ **Improved component detection**: Fixed issue with multiple types being listed
- ✅ **Enhanced error handling**: Better fallback for malformed LLM responses
- ✅ **Automatic syntax correction**: C-style comments and invalid syntax auto-fixed
- ✅ **Better type selection**: Ensures components have single, specific types

## Limitations

- **Maximum 100 pages per processing run**: This limit ensures optimal processing time and prevents memory issues. For larger documents, use the `--start` and `--end` flags to process specific sections.
- **Network Detection Accuracy**: Detection confidence varies based on diagram complexity and clarity. Adjust the `--confidence` threshold as needed.
- **Model Requirements**: Network detection requires vision-capable models (e.g., nanonets-ocr-s, qwen2.5vl, llava)
- **Timeout Behavior**: Operations that exceed the timeout will fall back to simpler processing methods
