Metadata-Version: 2.4
Name: indicflow
Version: 1.0.0
Summary: Analyse documents in Indian and Urdu languages using Gemini AI
Project-URL: Homepage, https://github.com/kraghavan/indicflow
Project-URL: Repository, https://github.com/kraghavan/indicflow
Project-URL: Issues, https://github.com/kraghavan/indicflow/issues
Author: Karthika Raghavan
License: MIT
License-File: LICENSE
Keywords: document-analysis,gemini,hindi,indic,nlp,ocr,pdf,tamil,telugu,translation,urdu
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: google-genai>=1.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pymupdf>=1.23.0
Requires-Dist: python-dotenv>=1.0.0
Provides-Extra: all
Requires-Dist: arabic-reshaper>=3.0.0; extra == 'all'
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'all'
Requires-Dist: pytest-cov>=5.0.0; extra == 'all'
Requires-Dist: pytest>=8.0.0; extra == 'all'
Requires-Dist: python-bidi>=0.4.2; extra == 'all'
Requires-Dist: python-docx>=1.1.0; extra == 'all'
Requires-Dist: python-pptx>=0.6.23; extra == 'all'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Provides-Extra: docx
Requires-Dist: python-docx>=1.1.0; extra == 'docx'
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'html'
Provides-Extra: pptx
Requires-Dist: python-pptx>=0.6.23; extra == 'pptx'
Provides-Extra: urdu
Requires-Dist: arabic-reshaper>=3.0.0; extra == 'urdu'
Requires-Dist: python-bidi>=0.4.2; extra == 'urdu'
Description-Content-Type: text/markdown

# 📄 indicflow

> Analyse documents in Indian and Urdu languages — extract, translate, and understand using Gemini AI

---

## Overview

indicflow is a single command-line tool with two modes:

**`--mode pipeline`** — Fast batch processing. Translates pages and flags keyword-relevant sections. Good for processing large folders of documents quickly.

**`--mode deep`** — In-depth single document analysis. Builds a word frequency table, enriches it with English translations, auto-selects the most meaningful nouns as keywords, and builds rich per-keyword context for every page those words appear on. Good for understanding an unfamiliar document or detecting bias and topic patterns.

Both modes support any document format and any of the supported languages.

---

## Supported Languages

| Language | Script | Notes |
|---|---|---|
| Hindi | Devanagari | |
| Tamil | Tamil | |
| Telugu | Telugu | |
| Kannada | Kannada | |
| Malayalam | Malayalam | |
| Bengali | Bengali | |
| Marathi | Devanagari | |
| Urdu | Nastaliq (Arabic) | RTL script — handled automatically |

---

## Supported Document Formats

| Format | Extension | Notes |
|---|---|---|
| PDF | `.pdf` | Text-based or scanned — auto-detected |
| Word | `.docx` | Requires `python-docx` |
| PowerPoint | `.pptx` | One page per slide. Requires `python-pptx` |
| Plain text | `.txt` `.md` | |
| HTML | `.html` `.htm` | Tags stripped automatically |
| CSV | `.csv` | Rows treated as plain text |
| Images | `.jpg` `.jpeg` `.png` `.webp` `.tiff` | Gemini Vision reads directly |

Scanned PDFs and image files with no selectable text are automatically routed to Gemini Vision — no extra OCR setup needed.

---

## Project Structure

```
indicflow/                     ← root of the GitHub repo
├── pyproject.toml             ← package metadata and dependencies
├── README.md
├── CHANGELOG.md               ← version history
├── CONTRIBUTING.md            ← how to contribute
├── PUBLISHING.md              ← release checklist
├── LICENSE
├── .gitignore
├── run_tests.sh               ← integration test suite (requires API key)
├── tests/                     ← unit tests (no API key needed)
│   ├── conftest.py
│   ├── helpers.py
│   ├── test_extractor.py
│   ├── test_gemini_client.py
│   ├── test_cli.py
│   ├── test_output_schema.py
│   └── fixtures/
│       └── test_outputs/      ← frozen JSON fixtures for schema tests
└── src/
    └── indicflow/             ← the installed Python package
        ├── __init__.py
        ├── __main__.py        ← enables: python -m indicflow
        ├── cli.py             ← entry point for both modes
        ├── extractor.py       ← universal document loader
        ├── gemini_client.py   ← all Gemini API calls
        └── models.py          ← Pydantic output schemas
```

---

## Setup

### Install from PyPI

```bash
# Core — PDF, TXT, MD, HTML, CSV, images
pip install indicflow

# Everything — adds Word, PowerPoint, and Urdu RTL support
pip install indicflow[all]

# Individual optional extras
pip install indicflow[docx]   # .docx support
pip install indicflow[pptx]   # .pptx support
pip install indicflow[html]   # better HTML extraction
pip install indicflow[urdu]   # Urdu RTL reshaping
```

### Add your API key

Create a `.env` file in whatever folder you will run `indicflow` from:

```
GEMINI_API_KEY=your_key_here
```

No quotes around the value. Both `GEMINI_API_KEY` and `GOOGLE_API_KEY` are accepted.
Get a free key at [aistudio.google.com/app/apikey](https://aistudio.google.com/app/apikey)

### Verify it works

```bash
indicflow --help
```

### For development (clone and edit)

```bash
git clone https://github.com/kraghavan/indicflow
cd indicflow
pip install -e ".[all]"
```

---

## All Flags

```
Shared (both modes):
  --input           Path to a file or folder (required)
  --language        Document language e.g. Hindi, Tamil, Urdu (required)
  --mode            pipeline or deep (default: deep)
  --translate       Include English translation per page / snippet
  --pages N M       Page range e.g. --pages 1 10 (PDF and PPTX only)
  --output          Output filename (default: pipeline_output.json / indicflow_output.json / *.csv)
  --output-format   json (default) or csv

Pipeline mode only:
  --keywords        Comma-separated words to scan for (English or native or mixed)
  --threshold       Relevance cutoff 0.0–1.0 (default: 0.15)

Deep mode only:
  --top-x           How many top nouns to use as auto-keywords (default: 10)
  --freq-min        Minimum word frequency to include in table (default: 2)
  --freq-top-n      Rows to show in printed frequency table (default: 50)
  --keyword-source  auto (default) or manual
  --keywords        Used when --keyword-source manual (comma-separated)
  --context-chars   Surrounding text per keyword occurrence (default: 400)
```

### Model override

By default indicflow uses `gemini-3-flash-preview`. To pin to a specific model, add to your `.env`:

```
INDICFLOW_MODEL=gemini-2.0-flash
```

Useful if the default model is experiencing high demand or you want to control costs.

---

## Mode 1 — Pipeline: Examples

### The simplest case — translate a single file

```bash
indicflow \
  --input court_notice.pdf \
  --language Tamil \
  --mode pipeline \
  --translate
```

> `python -m indicflow` works identically if you prefer that form.

---

### Translate a folder of documents in one run

```bash
indicflow \
  --input ./patient_records/ \
  --language Malayalam \
  --mode pipeline \
  --translate \
  --output translated_records.json
```

---

### Scan a document for keywords — same language

```bash
indicflow \
  --input property_dispute.pdf \
  --language Hindi \
  --mode pipeline \
  --keywords "जमीन,मालिक,न्याय,किरायेदार,अदालत" \
  --threshold 0.15 \
  --output dispute_scan.json
```

The `--threshold` controls sensitivity. `0.05` casts a wide net and flags more pages. `0.30` only flags pages with dense keyword matches.

---

### Scan using English keywords on a native-language document

```bash
indicflow \
  --input land_records.pdf \
  --language Hindi \
  --mode pipeline \
  --keywords "landlord,tenant,eviction,court order,ownership" \
  --threshold 0.20 \
  --output land_scan.json
```

---

### Mixed English and native keywords in one list

```bash
indicflow \
  --input case_file.pdf \
  --language Urdu \
  --mode pipeline \
  --translate \
  --keywords "justice,انصاف,landlord,زمیندار,کسان,evidence" \
  --threshold 0.15 \
  --output case_analysis.json
```

---

### Translate and keyword-scan a specific page range

```bash
indicflow \
  --input government_report.pdf \
  --language Telugu \
  --mode pipeline \
  --translate \
  --pages 10 25 \
  --keywords "budget,allocation,expenditure" \
  --output report_section.json
```

---

### Process a scanned Urdu document

```bash
indicflow \
  --input scanned_urdu_contract.pdf \
  --language Urdu \
  --mode pipeline \
  --translate \
  --keywords "معاہدہ,شرائط,ادائیگی" \
  --output contract_analysis.json
```

---

### Analyse a Word document

```bash
indicflow \
  --input minutes_of_meeting.docx \
  --language Hindi \
  --mode pipeline \
  --translate \
  --keywords "प्रस्ताव,निर्णय,बजट" \
  --output meeting_analysis.json
```

---

### Analyse an image or scanned photo

```bash
indicflow \
  --input letter_photo.jpg \
  --language Tamil \
  --mode pipeline \
  --translate \
  --output letter_analysis.json
```

---

### Pipeline output structure

```json
{
  "file": "property_dispute.pdf",
  "file_type": "pdf",
  "language": "Hindi",
  "pages_analysed": "1–12",
  "document_analysis": {
    "doc_type": "Legal Notice",
    "overall_summary": "A notice filed at the district court regarding a property ownership dispute.",
    "key_entities": ["Ram Kumar", "District Court Jodhpur", "Plot No. 44B"],
    "main_topics": ["property dispute", "land ownership", "eviction", "tenant rights"]
  },
  "keyword_summary": {
    "keywords": ["landlord", "tenant", "eviction", "court order"],
    "flagged_pages": [3, 5, 8, 11],
    "threshold": 0.20
  },
  "pages": [
    {
      "page": 3,
      "original_text": "जमींदार ने किरायेदार को...",
      "translated_text": "The landlord issued a notice to the tenant demanding...",
      "page_summary": "Formal eviction notice citing non-payment of rent.",
      "key_points": ["Rent unpaid since April 2023", "Court hearing scheduled for 12 March"],
      "entities": ["Ram Kumar", "12 March 2024", "Plot No. 44B"],
      "keyword_result": {
        "semantic_score": 0.94,
        "matched_keywords": ["landlord → जमींदार", "eviction → बेदखली"],
        "context_note": "Page contains the core eviction demand with specific legal language.",
        "flagged": true
      }
    }
  ]
}
```

---

### Pipeline CSV output

```bash
indicflow \
  --input property_dispute.pdf \
  --language Hindi \
  --mode pipeline \
  --translate \
  --keywords "landlord,tenant,eviction" \
  --output-format csv \
  --output dispute_results.csv
```

| Column | Description |
|---|---|
| `file` | Source filename |
| `language` | Document language |
| `doc_type` | Detected document type |
| `overall_summary` | Document-level summary |
| `main_topics` | Pipe-separated list of topics |
| `page` | Page number |
| `original_text` | First 200 chars of original text |
| `translated_text` | Full English translation (if `--translate`) |
| `page_summary` | 1–2 sentence page summary |
| `key_points` | Pipe-separated key points |
| `entities` | Pipe-separated names, dates, places |
| `kw_flagged` | true / false |
| `kw_semantic_score` | 0.0–1.0 relevance score |
| `kw_matched` | Pipe-separated matched keywords |
| `kw_context_note` | Why this page was flagged |
| `kw_mode` | cross-language / same-script |

> Written as UTF-8 with BOM so Excel opens Devanagari, Nastaliq, and Tamil script correctly.

---

## Mode 2 — Deep: Examples

### The simplest case — auto top 10 keywords with translations

```bash
indicflow \
  --input folk_tale.pdf \
  --language Hindi \
  --mode deep
```

The frequency table is printed in the terminal with English translations alongside each word.

---

### See more keyword candidates before committing

```bash
# Step 1: run with top-x 25 to see what's available
indicflow \
  --input report.pdf \
  --language Urdu \
  --mode deep \
  --top-x 25 \
  --freq-top-n 25

# Step 2: pick from the table and run with manual keywords
indicflow \
  --input report.pdf \
  --language Urdu \
  --mode deep \
  --keyword-source manual \
  --keywords "کسان,زمیندار,انصاف,موہر,پنچایت"
```

---

### Control the frequency table display

```bash
indicflow \
  --input annual_report.pdf \
  --language Tamil \
  --mode deep \
  --freq-top-n 75 \
  --freq-min 3 \
  --top-x 12
```

---

### Manual keywords — native script only

```bash
indicflow \
  --input court_record.pdf \
  --language Hindi \
  --mode deep \
  --keyword-source manual \
  --keywords "किसान,जमींदार,न्याय,कलश,पंचायत,राजा,लगान"
```

---

### Manual keywords — English only on a native document

```bash
indicflow \
  --input case_file.pdf \
  --language Telugu \
  --mode deep \
  --keyword-source manual \
  --keywords "farmer,landlord,justice,ownership,dispute,village council"
```

---

### Manual keywords — mixed English and native

```bash
indicflow \
  --input land_case.pdf \
  --language Urdu \
  --mode deep \
  --keyword-source manual \
  --keywords "justice,انصاف,landlord,زمیندار,کسان,power,ظلم,evidence"
```

---

### Include English translation of every snippet

```bash
indicflow \
  --input legal_brief.pdf \
  --language Hindi \
  --mode deep \
  --translate \
  --top-x 10 \
  --output brief_deep.json
```

---

### Wider context window per occurrence

```bash
indicflow \
  --input dense_report.pdf \
  --language Bengali \
  --mode deep \
  --context-chars 700 \
  --top-x 8 \
  --translate
```

---

### Analyse a specific section of a large document

```bash
indicflow \
  --input government_inquiry.pdf \
  --language Hindi \
  --mode deep \
  --pages 40 60 \
  --top-x 15 \
  --translate \
  --output inquiry_section.json
```

---

### Analyse a PowerPoint presentation

```bash
indicflow \
  --input budget_presentation.pptx \
  --language Telugu \
  --mode deep \
  --translate \
  --top-x 10 \
  --output budget_analysis.json
```

---

### Analyse a Word document

```bash
indicflow \
  --input medical_report.docx \
  --language Malayalam \
  --mode deep \
  --keyword-source manual \
  --keywords "diagnosis,treatment,patient,medication,hospital" \
  --translate \
  --output medical_deep.json
```

---

### Analyse a scanned image

```bash
indicflow \
  --input handwritten_letter.jpg \
  --language Urdu \
  --mode deep \
  --translate \
  --top-x 8 \
  --output letter_deep.json
```

---

### Deep mode output structure

```json
{
  "file": "folk_tale.pdf",
  "file_type": "pdf",
  "language": "Urdu",
  "pages_analysed": "1–5",
  "keyword_source": "auto",
  "keywords_used": ["کسان", "زمیندار", "انصاف", "موہر", "پنچایت"],
  "document_analysis": {
    "doc_type": "Folk Tale — Rajasthani",
    "overall_summary": "A Rajasthani folk tale about an honest farmer who discovers seven pots of gold coins buried in his landlord's field.",
    "key_entities": ["Vijaydan Detha", "NCERT", "Rajasthan"],
    "main_topics": ["land ownership", "moral integrity", "power and justice"]
  },
  "word_frequency": [
    {"rank": 1, "word": "کسان", "frequency": 18}
  ],
  "enriched_frequency": [
    {"word": "کسان", "frequency": 18, "english": "farmer", "pos": "noun"}
  ],
  "keyword_contexts": [
    {
      "keyword": "کسان",
      "total_pages_found": 3,
      "total_occurrences": 18,
      "page_contexts": [
        {
          "page": 1,
          "occurrence_count": 6,
          "translated_snippets": ["The farmer had made many requests not to pay rent."],
          "how_keyword_is_used": "Farmer positioned as moral centre but structurally powerless.",
          "sentiment": "mixed",
          "entities_nearby": ["landlord", "gold coins", "village council"],
          "topic_in_context": "Power imbalance between tenant farmer and landowner",
          "bias_indicator": "Farmer consistently described in subordinate terms.",
          "notable_quote": "کسان نے لگان نہ دینے کی خاطر بہت منتیں کیں"
        }
      ]
    }
  ]
}
```

---

### Deep CSV output

```bash
indicflow \
  --input folk_tale.pdf \
  --language Urdu \
  --mode deep \
  --top-x 10 \
  --translate \
  --output-format csv \
  --output folk_tale_analysis.csv
```

| Column | Description |
|---|---|
| `file` | Source filename |
| `keyword` | The keyword in original script |
| `english` | English translation of the keyword |
| `pos` | Part of speech (noun / proper_noun / adjective / verb) |
| `frequency` | Total occurrences in the document |
| `page` | Page number of this occurrence |
| `occurrence_count` | Times the keyword appears on this page |
| `how_keyword_is_used` | Gemini's analysis of the word's role |
| `sentiment` | positive / negative / neutral / mixed |
| `topic_in_context` | Main theme of this passage |
| `bias_indicator` | Any framing or power imbalance detected, or "none" |
| `notable_quote` | Most interesting sentence containing this keyword |
| `entities_nearby` | Pipe-separated names, places, concepts nearby |
| `translated_snippets` | Pipe-separated English translations of surrounding text |

> Multi-value fields use ` | ` as the separator so they stay in a single cell.

---

## Cross-Language Keyword Scanning

Both modes support mixing English and native-script keywords freely.

```bash
--keywords "gold coins,landlord,justice,किसान,پنچایت,زمیندار"
```

```
"gold coins" → matches मोहर, موہر, सोना automatically
"landlord"   → matches जमींदार, زمیندار
"justice"    → matches न्याय, انصاف
"किसान"      → matches "farmer" and کسان semantically
```

---

## Cost Estimate

Uses Gemini 3 Flash by default. Override via `INDICFLOW_MODEL` in `.env` if needed.

| Mode | Volume | Approx. Cost |
|---|---|---|
| pipeline — translate only | 100 pages | ~$0.08 |
| pipeline — with keywords | 100 pages | ~$0.10 |
| deep — 10 auto keywords, 5 pages | per run | ~$0.04 |
| deep — enrichment + translation | per run | ~$0.01 |

Free tier covers personal and small-scale use comfortably.

---

## Troubleshooting

**"API key not found"**
→ Check `.env` exists in the same folder and contains `GEMINI_API_KEY=your_key_here` with no quotes.

**503 UNAVAILABLE — model experiencing high demand**
→ Gemini 3 Flash is new and occasionally hits capacity spikes. The tool retries automatically. If it persists, pin to a stable model: add `INDICFLOW_MODEL=gemini-2.0-flash` to your `.env`.

**Scanned PDF or image shows no text**
→ Normal — the script switches to Gemini Vision automatically. Just let it run.

**Frequency table still showing prepositions after enrichment**
→ Switch to `--keyword-source manual` and list only the words you want.

**`--top-x 10` not giving useful keywords**
→ Run with `--top-x 25` first to see more candidates, then switch to manual mode.

**Keywords not matching across languages**
→ Include at least one keyword in a different script from the document.

**`python-docx` or `python-pptx` not found**
→ `pip install indicflow[docx]` or `pip install indicflow[pptx]`.

**Very large documents are slow or hit rate limits**
→ Use `--pages N M` to process a section at a time.

---

## Built With

- [PyMuPDF](https://pymupdf.readthedocs.io/) — PDF text extraction
- [Gemini 3 Flash](https://deepmind.google/models/gemini/flash/) — translation, enrichment, context analysis, semantic scoring
- [google-genai](https://pypi.org/project/google-genai/) — official Gemini Python SDK
- [Pydantic](https://docs.pydantic.dev/) — output schemas
- [python-docx](https://python-docx.readthedocs.io/) — Word document support
- [python-pptx](https://python-pptx.readthedocs.io/) — PowerPoint support
- [arabic-reshaper](https://pypi.org/project/arabic-reshaper/) + [python-bidi](https://pypi.org/project/python-bidi/) — Urdu RTL reshaping

---

> Built for analysing Indian and Urdu language documents with care for script accuracy, semantic depth, and context quality.
