Metadata-Version: 2.4
Name: aikosh
Version: 2.0.0
Summary: AIKosh unified SDK: model registry, download, and inference pipeline.
Author: AIKosh SDK contributors
License: Apache-2.0
Project-URL: Homepage, https://aikosh.indiaai.gov.in/home
Keywords: aikosh,indiaai,datasets,machine-learning,inference,pipeline
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: httpx<1,>=0.27
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: tqdm>=4.66
Requires-Dist: python-dotenv>=1.0
Provides-Extra: pipeline
Requires-Dist: transformers<5.0,>=4.53; extra == "pipeline"
Requires-Dist: torch>=2.4; extra == "pipeline"
Requires-Dist: accelerate>=0.30; extra == "pipeline"
Requires-Dist: peft>=0.13; extra == "pipeline"
Requires-Dist: safetensors>=0.4; extra == "pipeline"
Requires-Dist: sentencepiece>=0.2; extra == "pipeline"
Requires-Dist: huggingface_hub[hf_xet]; extra == "pipeline"
Requires-Dist: onnxruntime; extra == "pipeline"
Requires-Dist: numpy; extra == "pipeline"
Requires-Dist: soundfile; extra == "pipeline"
Requires-Dist: llama-cpp-python; extra == "pipeline"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"

# AIKosh SDK (Python)

Open-source Python SDK for working with the AIKosh platform: **discover datasets and models, inspect files, download assets, and run model inference** using the Pipeline API.

## What is AIKosh SDK?

AIKosh SDK is a **developer-friendly Python library** that wraps the AIKosh platform's APIs into **simple, stable functions** you can call from notebooks, scripts, and applications.

It's designed to feel like an "ML developer tool" library (similar in spirit to libraries like `transformers`), where common workflows—search, browse files, download, inference—are one import away.

## Why use an SDK (instead of calling APIs directly)?

Using an SDK helps developers by providing:

- **Simpler usage**: no manual URL construction, headers, or response parsing in every script.
- **Consistent patterns**: same function shapes for datasets and models (`list_directory`, `list_files`, `get_metadata`, `download`).
- **Safer downloads**: automatically fetches fresh temporary URLs and streams downloads to disk with automatic retry on failures.
- **Inference pipeline**: run AI models for NER, text generation, translation, summarization, embeddings, fill-mask, and text-to-speech with a single function call.
- **One place to evolve**: when the backend evolves, updating the SDK updates every downstream user.

## What this SDK aspires to help you build

Over time, the goal is to make it easy to build:

- **Repeatable data/model pipelines**: programmatic discovery + download for training/evaluation.
- **Dataset/model exploration tools**: list and traverse file trees for validation and QA.
- **AI applications**: run inference on AIKosh models, HuggingFace models, and local models.
- **Automation**: integrate AIKosh assets into CI workflows and internal platforms.

## Install

### Default (core download SDK only)

Includes dataset/model discovery, metadata, and download functionality:

```bash
pip install aikosh
```

### With Pipeline (inference support)

Adds full model inference support — HuggingFace, ONNX, and more backends:

```bash
pip install aikosh[pipeline]
```

This installs other required dependencies for model inference

> **Note:** The Pipeline API (`aikosh.pipeline(...)`) requires `aikosh[pipeline]`.

### For contributors / local development

```bash
pip install -e ".[dev]"
```

## Configuration

### API key

Creating and Managing Your API Key:

The API key feature on AIKosh empowers you to securely access and integrate platform datasets into your applications and workflows. By generating a personal API key, you can automate data retrieval, build custom analytical pipelines, and programmatically interact with the platform's resources.

Steps to be followed:
1. For creation of API key, login to AIKosh platform, click on My profile on top right corner, and go on Account settings.
2. Click on Create API key. A modal will show the Unique API key. Users may copy and store it as the unique API key will only be created once. Copy and securely store the key, as it will be used in your application headers for authenticated requests.
3. Once the API is generated, the same is shown in encrypted format which can be generated before creation of the new key.

For reference [Click Here](https://aikosh.indiaai.gov.in/public/files/selfcare/uploads-b7ba741d-ca01-426b-b539-6fea3e01d98c/User_Manual_AIKosh.pdf#page=41)

Option A — environment variable:

```python
import os
os.environ["AIKOSH_API_KEY"] = "YOUR_KEY"
```

Option B — in code:

```python
import aikosh
aikosh.set_api_key("YOUR_KEY")
```

(`AIKOSH_ACCESS_KEY` is also supported.)

## Asset identifiers (`id`)

List and metadata responses expose each dataset or model under the **`id`** field. Use that value everywhere the SDK expects an **`identifier`** (or the first argument to domain helpers like `list_files`).

```python
import aikosh

out = aikosh.list_directory("dataset", filters={"page": 1, "size": 10, "accessScope": "all"})
items = out["data"]["items"]  # shape depends on API; each item has "id"
dataset_id = items[0]["id"]

out = aikosh.list_directory("model", filters={"page": 1, "size": 10, "accessScope": "all"})
model_id = out["data"]["items"][0]["id"]
```

Do not use human-readable slugs where the API expects the platform **`id`**.

> **Note:** `"accessScope"` defaults to `"permitted"` (shows only open datasets/models). Use `{"accessScope": "all"}` to see the exhaustive list.

## Download request parameters

| Parameter | Role |
|-----------|------|
| `identifier` | Dataset or model **`id`** from list/metadata |
| `type` | `"dataset"` or `"model"` |
| `destination_path` | **Local** folder or file path where the download is saved |
| `file_path` | **Remote** file path inside the asset (single-file download) |
| `directory_path` | **Remote** folder/path inside the asset (can be combined with `filename`) |
| `filename` | Optional local output name; with `directory_path`, also joins the remote path |
| `version_id` | Optional version **`id`** when the API supports multiple versions |
| `max_workers` | Batch downloads only: parallel workers (default **4**, maximum **4**) |

**`directory_path` is not a local save path** — use **`destination_path`** for that.

## Quickstart

### 1) Check connectivity

```python
import aikosh
print(aikosh.ping())  # dataset filters endpoint
```

### 2) Discover available functions

```python
import aikosh
aikosh.list_functions()
# Returns: {"aikosh": {...}, "aikosh.datasets": {...}, "aikosh.models": {...}}
```

### 3) Filter master (codes for list filters)

```python
import aikosh

aikosh.get_datasets_filter_info()
# Returns: {"status": "success", "message": "filters endpoint reachable",
#   "data": {
#     "organisationList": [{"id":..., "name":...}, ...],
#     "sectorsList": [{"id":..., "name":...}, ...],
#     "licensesList": [{"id":..., "name":...}, ...],
#     "datasetTypesList": [{"id":..., "name":...}, ...]
#   }
# }

aikosh.get_models_filter_info()
# Returns: {"status": "success", "message": "filters endpoint reachable",
#   "data": {
#     "organisationList": [{"id":..., "name":...}, ...],
#     "sectorsList": [{"id":..., "name":...}, ...],
#     "licensesList": [{"id":..., "name":...}, ...],
#     "modelTypesList": [{"id":..., "name":...}, ...]
#   }
# }
```

### 4) List datasets or models

```python
import aikosh

out = aikosh.list_directory(
    "dataset",
    filters={"page": 1, "size": 20, "keyword": "sanskrit", "accessScope": "all"},
)
print(out["data"])

out = aikosh.list_directory(
    "model",
    filters={"page": 1, "size": 20, "keyword": "Bhashini", "modelType": [374, 375], "accessScope": "all"},
)
print(out["data"])

# If filters match nothing, check the SDK message (status stays "success"):
if out.get("message"):
    print(out["message"])
```

> **Note:** `"accessScope"` defaults to `"permitted"`. Use `{"accessScope": "all"}` for full listing.

#### Dataset list filters

```python
out = aikosh.list_directory(
    "dataset",
    filters={
        "page": 1,
        "size": 20,
        "license": [213, 214],
        "sector": [3228, 209],
        "fileFormat": ["csv", "json"],
        "versionScore": 3,
        "keyword": "Krishi",
        "accessScope": "all",
    },
)

out = aikosh.list_directory(
    "model",
    filters={
       "page": 1,
       "size": 20,
       "license": [212,7084],
       "sector": [190,210],
       "fileExtensions": ["csv","json"],
       "modelType": [3160, 2624],
       "keyword": "Bhashini",
    },
)
print(out)
```

### 5) Get metadata (datasets and models)

One function with `type` set to `"dataset"` or `"model"`:

```python
import aikosh

dataset_id = "PUT_DATASET_ID_HERE"  # from list response item["id"]
model_id = "PUT_MODEL_ID_HERE"

print(aikosh.get_metadata("dataset", dataset_id)["data"])
print(aikosh.get_metadata("model", model_id)["data"])
```

Aliases: `aikosh.get_dataset_metadata(dataset_id)` and `aikosh.get_model_metadata(model_id)`.

### 6) List files (datasets and models)

`directory_path` in **filters** is the **remote** folder inside the asset (`""` for root).

```python
import aikosh

dataset_id = "PUT_DATASET_ID_HERE"
model_id = "PUT_MODEL_ID_HERE"

aikosh.list_files(
    "dataset",
    dataset_id,
    filters={"directory_path": "", "page": 1, "limit": 50},
)

aikosh.list_files(
    "model",
    model_id,
    filters={"directory_path": "", "page": 1, "limit": 50},
)
```

Domain shortcuts: `aikosh.datasets.list_files(dataset_id, ...)` and `aikosh.models.list_files(model_id, ...)`.

### 7) Download

#### Whole dataset or model

```python
import aikosh

dataset_id = "PUT_DATASET_ID_HERE"
out = aikosh.download(
    {
        "identifier": dataset_id,
        "type": "dataset",
        "destination_path": "./downloads",
    }
)

model_id = "PUT_MODEL_ID_HERE"
out = aikosh.download(
    {
        "identifier": model_id,
        "type": "model",
        "destination_path": "./downloads/models",
        # "version_id": "OPTIONAL_VERSION_ID",
    }
)
```

#### Single file (remote path + local destination)

```python
out = aikosh.download(
    {
        "identifier": dataset_id,
        "type": "dataset",
        "file_path": "documents/report.pdf",
        "destination_path": "./downloads/files",
        "filename": "report_copy.pdf",
    }
)

# Or remote folder + file name
out = aikosh.download(
    {
        "identifier": model_id,
        "type": "model",
        "directory_path": "weights/",
        "filename": "model.bin",
        "destination_path": "./downloads/models/files",
    }
)
```

#### Batch download

Pass a **list** of download request dicts. Concurrency is controlled with `max_workers` (default **4**; values above **4** are capped at **4**).

```python
out = aikosh.download(
    [
        {"identifier": "DATASET_ID_1", "type": "dataset", "destination_path": "./downloads"},
        {"identifier": "DATASET_ID_2", "type": "dataset", "destination_path": "./downloads"},
    ],
    max_workers=4,  # optional; maximum allowed is 4
)
print(out["status"])  # success | partial_success | failed
print(out["items"])
```

---

## Pipeline API (Model Inference)

The **Pipeline API** allows you to run inference on AI models for a variety of tasks. It supports models from AIKosh registry, HuggingFace Hub, and your local directories.

### Ways to Run Inference

---

#### Way 1: Local Model (Folder Path or ZIP)

Use a model you have already downloaded to your local machine. The pipeline auto-detects the model type — no configuration needed.

```python
import aikosh

# Option A: Local folder
pipe = aikosh.pipeline(model="./downloads/models/IndicNER/IndicNER")
result = pipe("Narendra Modi visited New Delhi yesterday.")
print(result["output"])

# Option B: ZIP file (auto-extracted on first use)
pipe = aikosh.pipeline(model="./downloads/models/IndicNER.zip")
result = pipe("Apple Inc. was founded in California.")
print(result["output"])

# Option C: Explicit task override
pipe = aikosh.pipeline(
    model="./downloads/models/my_translation_model",
    task="translation"
)
result = pipe("translate English to Hindi: How are you?")
print(result["output"])
```

**How it works:**
- SDK inspects the model folder's `config.json` to auto-detect architecture and task
- ZIP files are automatically extracted before loading
- Once registered, subsequent calls use the cache instantly (`<1ms`)

---

#### Way 2: HuggingFace Hub Model (by ID)

Use any model directly from HuggingFace Hub by providing the `org/model-name` ID. Requires `allow_external_download=True`.

```python
import aikosh

# Text generation
pipe = aikosh.pipeline(
    model="google/flan-t5-base",
    allow_external_download=True,
)
result = pipe("What is the capital of India?")
print(result["output"])

# NER
pipe = aikosh.pipeline(
    model="dslim/bert-base-NER",
    allow_external_download=True,
    task="ner"
)
result = pipe("Tata Motors is headquartered in Mumbai, India.")
print(result["output"])

# Translation
pipe = aikosh.pipeline(
    model="krutrim-ai-labs/Krutrim-Translate",
    allow_external_download=True,
    task="translation"
)
result = pipe("translate English to Hindi: Welcome to India.")
print(result["output"])

# Embeddings
pipe = aikosh.pipeline(
    model="sentence-transformers/all-MiniLM-L6-v2",
    allow_external_download=True,
    task="embedding"
)
result = pipe("India is a diverse nation.")
print(result["output"])   # numpy array of shape (384,)
```

**How it works:**
- SDK fetches `config.json` from HuggingFace to detect architecture
- Model downloads to HuggingFace's local cache
- Automatically registered so future calls are instant

---

#### Way 3: AIKosh Portal Model (by UUID)

Use models published on the AIKosh platform directly by their UUID. Requires an API key and `allow_external_download=True`.

```python
import aikosh

# Set API key first
aikosh.set_api_key("YOUR_AIKOSH_API_KEY")

# Load model from AIKosh registry
pipe = aikosh.pipeline(
    model="8c5289a4-2a2b-457f-8a27-6ac156c1b013",   # UUID from AIKosh portal
    source="aikosh",
    allow_external_download=True,
    destination_path="./downloads/models"             # where to save the model
)
result = pipe("Classify this Hindi text")
print(result["output"])
```

**For externally hosted models** (model hosted outside AIKosh, e.g. on another platform):

```python
# The download response will indicate external hosting
out = aikosh.download({
    "identifier": "EXTERNAL_MODEL_UUID",
    "type": "model",
    "destination_path": "./downloads/models"
})

# Response for externally hosted model:
# {
#     "status": "success",
#     "type": "model",
#     "identifier": "...",
#     "info": {
#         "externalUrl": "https://external-source.com/model",
#         "source": "external-source.com",
#         "msg": "Model is onboarded from the external source. Kindly redirect to the mentioned URL"
#     }
# }

```

**How it works:**
- SDK validates the UUID format
- Downloads the model files to `destination_path`
- Auto-detects and registers the model
- If externally hosted: returns redirect info (must download manually)

---

#### Way 4: Remote Adapter (HuggingFace Hub Model ID)

```python
import aikosh
```

Create a `.env` file in your project root and add your HuggingFace access token (select **read** access when generating the token on HuggingFace):

```bash
# .env
HF_TOKEN="your_hf_token_here"
```

## Backend Selection for Hugging Face Models

When a Hugging Face model ID (for example, `Qwen/Qwen2.5-0.5B-Instruct`) is passed to `pipeline()`, the SDK automatically selects the appropriate backend based on the model registry and the pipeline arguments.

### Case 1: Hugging Face model ID is **NOT** present in `model_registry.json`

- **Default (`backend="auto"`)**
  - The SDK automatically routes the request to the **Hugging Face Remote** backend.
  - No model weights are downloaded locally.
  - Inference is performed through the Hugging Face remote inference service.

- **`backend="huggingface_remote"`**
  - Explicitly uses the Hugging Face Remote backend.
  - No local model download.

- **`allow_external_download=True`**
  - The SDK downloads the model from the Hugging Face Hub, registers it in memory, and performs inference using the local Hugging Face backend.

- **`backend="huggingface"`**
  - Explicitly uses the Hugging Face backend.
  - model download local.  

### Case 2: Hugging Face model ID is already present in `model_registry.json`

The SDK first checks `model_registry.json`. If the model is already registered, the registry configuration is used by default.

#### Registered with `backend: "huggingface_remote"`

If the model is registered with the **Hugging Face Remote** backend (for example, `meta-llama/Llama-3.3-70B-Instruct`):

**Default (no backend specified)**

```python
pipe = aikosh.pipeline(
    task="text-generation",
    model="meta-llama/Llama-3.3-70B-Instruct",
)
```

- Uses the **Hugging Face Remote** backend defined in `model_registry.json`.

**With `allow_external_download=True`**

```python
pipe = aikosh.pipeline(
    task="text-generation",
    model="meta-llama/Llama-3.3-70B-Instruct",
    allow_external_download=True,
)
```

- Still uses the **Hugging Face Remote** backend.
- `allow_external_download=True` does **not** override permanently registered remote models.

**With `backend="huggingface_remote"`**

```python
pipe = aikosh.pipeline(
    task="text-generation",
    model="meta-llama/Llama-3.3-70B-Instruct",
    backend="huggingface_remote",
)
```

- Explicitly uses the **Hugging Face Remote** backend.

**With `backend="huggingface"`**

```python
pipe = aikosh.pipeline(
    task="text-generation",
    model="meta-llama/Llama-3.3-70B-Instruct",
    backend="huggingface",
)
```

- The SDK attempts to use the local Hugging Face backend.
- Since the model is registered as a **remote-only** model, local model weights are not available.
- A `ModelLoadError` is raised, indicating that the model cannot be loaded locally using the Hugging Face backend.

> **Note:** For models permanently registered with `backend: "huggingface_remote"` in `model_registry.json`, the registry configuration takes precedence. `allow_external_download=True` does not change these models to local inference.

## Examples

### Example 1: Default (`backend="auto"`)

```python

pipe = aikosh.pipeline(
    task="text-generation",
    model="Qwen/Qwen2.5-7B-Instruct",
)

result = pipe("Summarise deep learning in one sentence.")

print(f"Output : {result['text']}")
print(f"Backend: {result['metadata'].get('backend')}")
```

**Output**

```text
Backend: huggingface_remote
```

---

### Example 2: Explicit Remote Backend

You can explicitly request the Hugging Face Remote backend.

```python

pipe = aikosh.pipeline(
    task="text-generation",
    model="Qwen/Qwen2.5-7B-Instruct",
    backend="huggingface_remote",
)

result = pipe("Summarise deep learning in one sentence.")

print(f"Output : {result['text']}")
print(f"Backend: {result['metadata'].get('backend')}")
```

**Output**

```text
Backend: huggingface_remote
```

---

### Example 3: Local Download from Hugging Face Hub

Setting `allow_external_download=True` downloads the model from the Hugging Face Hub and performs **local inference**.

```python

pipe = aikosh.pipeline(
    task="text-generation",
    model="Qwen/Qwen2.5-7B-Instruct",
    allow_external_download=True,
)

result = pipe("Summarise deep learning in one sentence.")

print(f"Output : {result['text']}")
print(f"Backend: {result['metadata'].get('backend')}")
```

**Output**

```text
Backend: huggingface
```

> **Note:** `allow_external_download=True` downloads the model weights from the Hugging Face Hub and uses the local Hugging Face backend. It does **not** use the Hugging Face Remote backend.

### Basic Usage

```python
import aikosh

# Create a pipeline (task auto-detected from model)
pipe = aikosh.pipeline(model="google-t5/t5-small", allow_external_download=True)

# Run inference
result = pipe("translate English to German: Hello, how are you?")
print(result["output"])   # translated text
print(result["task"])     # "text2text-generation"
```

### Pipeline Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `task` | `None` (auto-detect) | Task type: `"text-generation"`, `"ner"`, `"translation"`, `"summarization"`, `"fill-mask"`, `"embedding"`, `"text-to-speech"` |
| `model` | `"google-t5/t5-small"` | Model name, HuggingFace ID, local path, ZIP file path, or AIKosh UUID |
| `source` | `"auto"` | Model source: `"auto"`, `"aikosh"`, `"huggingface"`, `"local"`, `"huggingface_remote"` |
| `allow_external_download` | `False` | Allow downloading from HuggingFace Hub or AIKosh |
| `destination_path` | `"./downloads/models"` | Local path for downloaded AIKosh models |
| `backend` | `"auto"` | Backend: `"auto"`, `"huggingface"`, `"onnx"` ,`"huggingface_remote"` |
| `device` | `"auto"` | Device: `"auto"`, `"cpu"`, `"cuda"` |
| `max_new_tokens` | `128` | Max tokens to generate |
| `temperature` | `0.7` | Sampling temperature |
| `top_p` | `0.95` | Top-p sampling |
| `trust_remote_code` | `False` | Allow models that ship custom modeling code not in standard `transformers` (e.g. MiniMax, Falcon, Phi-3, DeepSeek). Set `True` only for repos you trust. |
| `debug` | `False` | Include prompt preview in response metadata |

### Pipeline Response

All pipelines return a dictionary with the following fields:

```python
{
    "task": "text-generation",       # task type
    "model": "google-t5/t5-small",   # model used
    "output": "generated text",       # main result
    "output_type": "text",           # "text" | "audio" | "tokens" | "embeddings"
    "usage": {},                      # token counts etc.
    "artifacts": [],                  # saved file paths (e.g. .wav for TTS)
    "metadata": {},                   # timing, debug info
    "text": "generated text"          # legacy alias for output
}
```

### Supported Tasks

#### Text Generation

```python
import aikosh

pipe = aikosh.pipeline(
    model="google-t5/t5-small",
    allow_external_download=True,
    task="text2text-generation"
)
result = pipe("summarize: The quick brown fox jumps over the lazy dog.")
print(result["output"])
```

#### Summarization

```python
pipe = aikosh.pipeline(
    model="google-t5/t5-small",
    allow_external_download=True,
    task="summarization"
)
result = pipe(long_text, max_length=150, min_length=40)
print(result["output"])
```

#### Translation

```python
pipe = aikosh.pipeline(
    model="google-t5/t5-small",
    allow_external_download=True,
    task="translation"
)
result = pipe("translate English to French: Where is the nearest hospital?")
print(result["output"])
```

#### Named Entity Recognition (NER)

```python
pipe = aikosh.pipeline(
    model="dslim/bert-base-NER",
    allow_external_download=True,
    task="ner"
)
result = pipe("Apple Inc. was founded by Steve Jobs in California.")
print(result["output"])
# [{"word": "Apple Inc.", "entity": "ORG", ...}, {"word": "Steve Jobs", "entity": "PER", ...}]
```

#### Fill-Mask

```python
pipe = aikosh.pipeline(
    model="distilbert-base-uncased",
    allow_external_download=True,
    task="fill-mask"
)
result = pipe("The capital of France is [MASK].")
print(result["output"])
# [{"token_str": "Paris", "score": 0.99, ...}]
```

#### Embeddings

```python
pipe = aikosh.pipeline(
    model="sentence-transformers/all-MiniLM-L6-v2",
    allow_external_download=True,
    task="embedding"
)
result = pipe("The weather is nice today.")
print(result["output"])        # numpy array of shape (384,)
print(result["output_type"])   # "embeddings"
```

#### Text-to-Speech

```python
pipe = aikosh.pipeline(
    model="microsoft/speecht5_tts",
    allow_external_download=True,
    task="text-to-speech",
)
result = pipe("Welcome to AIKosh, India's AI platform.")
print(result["artifacts"])   # [{"type": "audio_file", "path": "output.wav"}]
print(result["output_type"]) # "audio"
```

### Model Sources

#### HuggingFace Hub Models

Any model from HuggingFace Hub can be used with `allow_external_download=True`:

```python
pipe = aikosh.pipeline(
    model="google/flan-t5-base",          # HuggingFace model ID (org/model)
    allow_external_download=True,
    task="text2text-generation"
)
result = pipe("What is the capital of India?")
print(result["output"])
```

The SDK automatically:
1. Fetches the model's `config.json` from HuggingFace
2. Detects the architecture and task type
3. Registers it for future use (no re-detection on subsequent calls)

#### AIKosh Registry Models

Use a model's UUID from the AIKosh platform:

```python
import aikosh

aikosh.set_api_key("YOUR_API_KEY")

pipe = aikosh.pipeline(
    model="8c5289a4-2a2b-457f-8a27-6ac156c1b013",  # AIKosh model UUID
    source="aikosh",
    allow_external_download=True,
    destination_path="./downloads/models"
)
result = pipe("Classify this text")
print(result["output"])
```

#### Local Models (Directory or ZIP)

```python
# From a local directory
pipe = aikosh.pipeline(model="./my_local_model")
result = pipe("Input text here")

# From a ZIP file (auto-extracted)
pipe = aikosh.pipeline(model="./models/bert-model.zip")
result = pipe("The capital is [MASK].")
```

### Pre-registered Models

The following models are pre-registered and work without `allow_external_download=True` if already cached locally:

| Model | Task | Notes |
|-------|------|-------|
| `google-t5/t5-small` | text2text-generation, summarization, translation | Default model |
| `skylord/kisanSLM` | text-generation, chat | Agriculture assistant (PEFT) |
| `sentence-transformers/all-MiniLM-L6-v2` | embedding | 384-dim embeddings |
| `microsoft/speecht5_tts` | text-to-speech | Audio generation |
| `distilbert-base-uncased` | fill-mask | Masked language model |
| `distilgpt2` | text-generation | GPT-2 based generation |
| `google/flan-t5-small` | text-generation, translation, summarization | Instruction-tuned |
| `krutrim-ai-labs/Krutrim-Translate` | translation | Indic language translation |
| `ai4bharat/indictrans2-indic-indic-1B` | translation | Indic-to-Indic translation |
| `Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens` | text-generation | Bharat MiniGPT |

### Model Caching

The SDK uses an efficient caching system:
- Models are inspected and registered **only once**
- Subsequent calls with the same model use the cache instantly (`<1ms`)
- Local models are tracked by directory path
- HuggingFace models are cached after first download

```python
# First call: downloads + registers (~minutes)
pipe1 = aikosh.pipeline(model="google/flan-t5-base", allow_external_download=True)

# Second call: uses cache (instant)
pipe2 = aikosh.pipeline(model="google/flan-t5-base", allow_external_download=True)
```

### Enable Logging

To see detailed pipeline loading and inference information:

```python
import aikosh
aikosh.enable_logging()  # INFO level by default

# Custom level and format
import logging
aikosh.enable_logging(
    level=logging.DEBUG,
    format='[%(asctime)s] %(message)s'
)
```

---

## Download Reliability

The SDK includes built-in reliability features for downloads to handle intermittent server issues (common with government data portals like data.gov.in):

### Automatic Retry

Downloads automatically retry up to **3 times** with a **3-second delay** on:
- `500`, `502`, `503`, `504` HTTP errors
- Network timeouts

### Browser-Compatible Headers

Downloads use browser-like headers to avoid being blocked by servers that reject automated tools.

### Per-Component Timeouts

| Component | Timeout |
|-----------|---------|
| TCP connect | 30s |
| Read (between chunks) | 120s |
| Write | 30s |
| Connection pool | 10s |

---

## Modules (what to import)

- **`import aikosh`**: most users only need this (high-level journey functions + pipeline).
- **`import aikosh.datasets`**: dataset journey + raw HTTP helpers.
- **`import aikosh.models`**: model journey + raw HTTP helpers.
- **`from aikosh.datasets import api as ds_api`**: advanced usage (parsed `data` from HTTP).
- **`from aikosh.models import api as models_api`**: same for models.

## Reference: top-level package (`import aikosh`)

| Function | Typical use |
|----------|-------------|
| `set_api_key` / `set_access_key` | Store API key in-process (also reads env vars). |
| `get_access_key` | Read the configured key (if any). |
| `pipeline(model, task, ...)` | Create an inference pipeline for AI tasks. |
| `enable_logging(level, format)` | Enable aikosh pipeline logging output. |
| `get_metadata(type, identifier, ...)` | Metadata for datasets or models. |
| `get_dataset_metadata(identifier, ...)` | Same as `get_metadata("dataset", identifier, ...)`. |
| `get_model_metadata(identifier, ...)` | Same as `get_metadata("model", identifier, ...)`. |
| `list_directory(type, filters=..., ...)` | List datasets or models. |
| `list_files(type, identifier, filters=..., ...)` | List files inside a dataset or model. |
| `download(request, ..., max_workers=...)` | Download dataset or model (single dict or batch list). |
| `to_json(data, ...)` | Serialize nested structures to a JSON string. |
| `ping(...)` | Connectivity check (dataset filters endpoint). |
| `list_functions(...)` | List user-facing functions and one-line descriptions. |
| `get_datasets_filter_info()` | Dataset filter master (codes for list filters). |
| `get_models_filter_info()` | Model filter master (codes for list filters). |
| `__version__` | Installed package version string. |

## Reference: `aikosh.models`

### Journey (user-facing)

| Function | Purpose |
|----------|---------|
| `list_directory(filters=..., ...)` | List models (`page`, `size`, `license`, `sector`, `fileFormat`, `modelType`, `keyword`). |
| `get_metadata(model_id, ...)` | Model metadata by **`id`**. |
| `list_files(model_id, filters=..., ...)` | Remote file tree (`directory_path`, optional `version_id`, `page`, `limit`). |
| `download(request, ..., max_workers=...)` | Download whole model or one file (batch supported). |
| `ping(...)` | Model filters connectivity check. |
| `to_json(data, ...)` | JSON helper. |

### Low-level API

| Function | Purpose |
|----------|---------|
| `require_uuid_string(name, value)` | Validate identifier format before API calls. |
| `get_filters`, `list_models`, `get_model_metadata`, `list_file_details` | Raw HTTP wrappers. |
| `get_model_download_url`, `get_file_download_url`, `stream_download_url_to_path` | Presigned URLs and streaming to disk. |

## Reference: `aikosh.datasets`

### Journey

| Function | Purpose |
|----------|---------|
| `list_directory("dataset", filters=..., ...)` | List datasets. |
| `get_metadata("dataset", dataset_id, ...)` | Dataset metadata by **`id`**. |
| `get_dataset_metadata_journey(dataset_id, ...)` | Same as `get_metadata("dataset", ...)`. |
| `list_files(dataset_id, filters=..., ...)` | Remote file tree for a dataset. |
| `download(..., max_workers=...)` | Dataset downloads (single or batch). |
| `ping(...)` | Dataset filters connectivity check. |
| `to_json(...)` | JSON helper. |

### Low-level API

`get_filters`, `list_datasets`, `get_dataset_metadata`, `list_file_details`, `get_dataset_version_download_url`, `get_file_download_url`, `stream_download_url_to_path`, `require_uuid_string`.

## Notes and limitations

- **Use `id` from API responses** as `identifier` in download/metadata/list_files calls.
- **Batch downloads**: `max_workers` defaults to **4** and cannot exceed **4**.
- **Unified top-level APIs**: `list_directory`, `get_metadata`, `list_files`, and `download` all accept `type="dataset"` or `type="model"`.
- **Model-specific shortcuts**: `aikosh.models.list_files`, `aikosh.models.ping`, etc., when you prefer not to pass `type`.
- **Pipeline caching**: models are inspected once and cached; repeat calls are instant.
- **External downloads**: set `allow_external_download=True` for HuggingFace and AIKosh models.

## Troubleshooting

- **401 / Invalid API key**: re-check `AIKOSH_API_KEY` or `aikosh.set_api_key(...)`.
- **422 invalid id**: pass the **`id`** from `list_directory(...)` / metadata, not a slug or display name.
- **No results from list search**: if `list_directory` is called with filters (e.g. `keyword`) and nothing matches, the response includes a **`message`** field. Pagination-only calls (`page` / `size` alone) do not add this message.
- **Wrong download location**: use `destination_path` for local saves; `directory_path` is only for remote paths inside the asset.
- **Download timeout / 504**: the remote server (e.g. data.gov.in) is temporarily unavailable. The SDK retries 3 times automatically; if all fail, wait a few minutes and try again.
- **Pipeline: `allow_external_download` required**: set `allow_external_download=True` when loading HuggingFace or AIKosh models for the first time.
- **Pipeline: API key required**: call `aikosh.set_api_key(...)` before using models with `source="aikosh"`.
- **Pipeline: model not found**: for HuggingFace, use the format `"org/model-name"`; for local models, use a path like `"./my_model"`.

## License

Apache-2.0
