Metadata-Version: 2.4
Name: waveflowdb_client
Version: 1.0.1
Summary: VectorLake SDK — Deterministic backend engine powering agent workflows
Author-email: "agentanalytics.ai" <nitin@agentanalytics.ai>
License: MIT License
        
        Copyright (c) 2026 agentanalytics.ai
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://agentanalytics.ai
Project-URL: Documentation, https://www.agentanalytics.ai/docs/waveflow-db
Keywords: vector db,VECTOR QUERY LANGUAGE,waveflowdb,agentanalytics,VQL
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: numpy
Requires-Dist: tqdm
Dynamic: license-file

# WaveflowDB SDK — VectorLake Python Client v2.0.0

A Python SDK for interacting with **WaveflowDB** and performing **WaveQL (VQL) brace-based semantic retrieval**.

This SDK provides:
* Full `Config` management via constructor args, environment variables, or `.env` file
* Document ingestion — direct payload mode or batched filesystem mode
* Intelligent document sync with MD5-based change detection
* Resumable batch uploads with `start_from_batch` / `end_batch` range control
* Semantic search and top-matching-doc retrieval with hybrid filtering
* Namespace and document metadata queries
* Structured CSV logging for performance, errors, and skipped files
* Simple `{stem}_part{num}.txt` chunk naming for idempotent re-runs

---

## 📌 Overview

Vector Lake is an **agentic backend for enterprises to build AI products**, enabling:
* Natural-language structured filtering through **WaveQL (VQL)**
* Hybrid ranking (Filter + Semantic)
* Zero-schema ingestion (no JSON schemas required)
* SQL-like logical joins on raw text
* Automatic semantic fallback when filters are absent

---

## 🚀 Getting Started

### 1. Install Dependencies

```bash
pip install waveflowdb_client
```

Optional extras for richer file support:

```bash
pip install PyPDF2 python-docx python-dotenv tqdm
```

### 2. Configure API Credentials

The SDK resolves settings in this priority order: **constructor args → environment variables → `.env` file → hard-coded defaults**.

**Option A — `.env` file**

```
VECTOR_LAKE_API_KEY=your_api_key_here
VECTOR_LAKE_HOST=https://waveflow-analytics.com
VECTOR_LAKE_PORT=8030
VECTOR_LAKE_PATH=upload
VECTOR_LAKE_LOG_DIR=logs
VECTOR_LAKE_TIMEOUT=240
VECTOR_LAKE_MAX_RETRIES=2
VECTOR_LAKE_MAX_FILES_PER_BATCH=100
VECTOR_LAKE_MAX_BATCH_SIZE_MB=0.5
VECTOR_LAKE_ALLOWED_EXTENSIONS=txt,csv,json,jsonl,py,docx,pdf
USER_ID=your@email.com
NAMESPACE=your_namespace
```

**Option B — Constructor arguments**

```python
from waveflowdb_client import Config, VectorLakeClient

cfg = Config(
    api_key="your_api_key_here",
    host="https://waveflow-analytics.com",
    service_port=8030,
    vector_lake_path="/path/to/documents",
)
client = VectorLakeClient(cfg)
```

---

## 🗂️ Supported File Types

The allowed extension list is the **single source of truth** defined in `Config` (or `VECTOR_LAKE_ALLOWED_EXTENSIONS`). The default set is:

| Extension | Processing strategy |
|-----------|-------------------|
| `txt` | Plain text, paragraph-chunked |
| `py` | Plain text read, paragraph-chunked |
| `ipynb` | Code cells extracted, paragraph-chunked |
| `pdf` | Text extracted via PyPDF2, paragraph-chunked |
| `docx` | Text extracted via python-docx, paragraph-chunked |
| `csv` | Row-safe split; header preserved in every chunk |
| `jsonl` / `ndjson` | Record-safe split; each line validated as JSON |
| `json` | Kept atomic — never split |

To add or remove extensions without changing code, update `VECTOR_LAKE_ALLOWED_EXTENSIONS` in your `.env`.

---

## 📦 Chunk Naming

All chunks written to disk follow this simple convention:

```
{stem}_part{num}.txt
```

Examples — a PDF that splits into 3 parts:
```
electrolytes-test-report_part1.txt
electrolytes-test-report_part2.txt
electrolytes-test-report_part3.txt
```

All formats (pdf, docx, py, ipynb, txt, etc.) produce `.txt` chunk files. The `VersionedChunkName` model provides `build()` and `parse()` helpers and an `is_versioned()` static method. Chunks already on disk are **reused on re-run** — the pipeline is idempotent and safe to restart after a crash.

---

## 🔄 Document Sync — How Uploads Work

```
  YOUR FOLDER (vector_lake_path/)
  ┌─────────────────────────────────────────────┐
  │  report.pdf   notes.txt   data.csv  ...     │
  └─────────────────────────┬───────────────────┘
                             │
                    sync_documents()
                    classifies each file
                             │
          ┌──────────────────┼───────────────────┐
          ▼                  ▼                   ▼
     No chunk          Chunk exists,        Chunk exists,
      on disk           MD5 matches          MD5 differs
          │                  │                   │
        NEW              UNCHANGED             CHANGED
          │                  │                   │
      add_docs()           skip             refresh_docs()
          │                                      │
          └──────────────────┬───────────────────┘
                             ▼
                    Upload in batches
                    (resumable via start_from_batch)
```

`sync_documents()` is the **recommended upload method** for most use cases. Use `add_documents()` or `refresh_documents()` directly only when you are certain all files are brand-new or all already exist in the namespace.

### Resumable Batch Ranges

All upload methods support `start_from_batch` and `end_batch`. If a long run is interrupted, set `START_FROM` to the last successful batch + 1 and re-run:

```
  Batch 1 ✓  Batch 2 ✓  Batch 3 ✓  [CRASH]  Batch 4 ...  Batch 39
  ──────────────────────────────────────────────────────────────────
                                     ↑
                              start_from_batch=4
                              Re-run resumes here
```

---

## Search Modes — `flat` vs `flat_filter`

The two `search_type` values represent fundamentally different retrieval strategies. Both require `hybrid_filter=True`.

### `search_type="flat"` — Full Corpus Fusion

Both semantic search and hybrid filter run **independently across ALL documents** in the namespace. Their results are then combined by the proprietary fusion algorithm, which re-ranks everything into a single fused list.

```
  Your Query
      │
      ├─────────────────────────────────┐
      ▼                                 ▼
  Semantic Search               Hybrid Filter
  (ALL documents                (keyword scan
   in namespace)                 ALL documents)
      │                                 │
      │   ┌─────────────────────────────┘
      ▼   ▼
  Proprietary Fusion Algorithm
  (combines + re-ranks both result sets)
      │
      ▼
  ┌─────────────────────────────────────────────────┐
  │  Tier 1: matched BOTH  →  highest fused score   │
  │  Tier 2: filter match only                      │
  │  Tier 3: semantic match only (fallback)         │
  └─────────────────────────────────────────────────┘
```

**Use `flat` for:** exploratory search, maximum recall, when you want every document to have a chance to surface.

---

### `search_type="flat_filter"` — Filtered Semantic Search

The hybrid filter runs first as a **hard gate** — only documents that pass are eligible for semantic search. Semantic ranking then runs only over that filtered candidate set.

```
  Your Query
      │
      ▼
  Hybrid Filter
  (keyword scan ALL documents)
      │
      ▼
  Filtered candidate set
  (only docs that passed the filter)
      │
      ▼
  Semantic Search
  (only over filtered candidates)
      │
      ▼
  Final ranked results
  (higher precision, smaller result set)
```

**Use `flat_filter` for:** targeted retrieval, when filter criteria are strong, when you want tighter and more focused results.

---

### Quick Comparison

```
  ┌─────────────────┬────────────────────────────┬────────────────────────────┐
  │                 │           flat             │        flat_filter         │
  ├─────────────────┼────────────────────────────┼────────────────────────────┤
  │ Semantic scope  │ ALL documents              │ Filter-passing docs only   │
  │ Filter role     │ Scoring signal (boosts)    │ Hard gate (excludes)       │
  │ Result style    │ Fused, broad               │ Precise, narrower          │
  │ Best for        │ Exploratory search         │ Targeted retrieval         │
  │ hybrid_filter   │ Required (True)            │ Required (True)            │
  └─────────────────┴────────────────────────────┴────────────────────────────┘
```

---

## `starter_script.py` — Function Reference & Sample Outputs

`starter_script.py` is the ready-to-use launcher included with the SDK. Set your credentials at the top via `.env`, drop files into your upload folder, uncomment exactly one function in `__main__`, and run:

```bash
python starter_script.py
```

### Function Map

```
  starter_script.py
  │
  ├── UPLOAD ──────────────────────────────────────────────────────────────
  │   ├── run_add_direct()          Add new files via string content
  │   ├── run_add_path()            Add new files from disk (batched)
  │   ├── run_refresh_direct()      Update existing files via string content
  │   ├── run_refresh_path()        Update existing files from disk (batched)
  │   ├── run_sync()                Auto-classify & route (recommended)
  │   └── run_sync_dry_run()        Preview sync plan, no upload
  │
  ├── QUERY ───────────────────────────────────────────────────────────────
  │   ├── run_match_static(q)             flat search, static index
  │   ├── run_match_filtered(q)           flat_filter search, static index
  │   ├── run_match_dynamic(q)            flat search, temp docs
  │   └── run_match_dynamic_filtered(q)   flat_filter search, temp docs
  │
  └── INFO ────────────────────────────────────────────────────────────────
      ├── run_health()              Ping server
      ├── run_namespace_details()   Storage & quota info
      └── run_docs_info()           List all indexed documents
```

---

### `run_health()`

Pings the backend to confirm connectivity and that your namespace is reachable.

**Sample output:**
```
──────────────────────────────────────────────────────────────────────
  HEALTH CHECK
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "Processing time at server 0.00 ms",
    "docs": []
  },
  "message": "Health check successful",
  "status_code": 200
}
```

---

### `run_add_direct()`

Uploads brand-new documents by passing file names and content directly as strings — no files on disk needed.

**Sample output:**
```
────────────────────────────────────────────────────────────
  ▶  ADD_DOCUMENTS  (direct mode)
────────────────────────────────────────────────────────────
making request
   ✓  Done
────────────────────────────────────────────────────────────
──────────────────────────────────────────────────────────────────────
  ADD DOCUMENTS — direct
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "2 file(s) uploaded successfully",
    "docs": {
      "new_files": ["test2", "test1"],
      "updated_files": [],
      "failed_files": []
    }
  },
  "message": "2 file(s) uploaded successfully",
  "status_code": 200
}
```

---

### `run_add_path()`

Reads all supported files from `VECTOR_LAKE_PATH`, chunks them into `{stem}_part{num}.txt` files, and uploads in batches with a live progress bar. Use only when **all files are brand-new** to the namespace.

**Sample output:**
```
────────────────────────────────────────────────────────────────
  ▶  ADD_DOCS
────────────────────────────────────────────────────────────────
   Source files  : 152
   Chunks        : 164
   This run      : 39 batches
   Overall total : 39 batches
   Batch range   : #1 → #39
────────────────────────────────────────────────────────────────
INFO: add_docs — 152 files | 39 batches (overall: 39) | range #1 → #39
add_docs:   3%|████                    | 1/39 [00:22<14:00, 22.1s/batch]
...
────────────────────────────────────────────────────────────────
  ■  ADD_DOCS COMPLETE
────────────────────────────────────────────────────────────────
   ✓  All batches succeeded
   Batch range   : #1 → #39
   Succeeded : 39/39
   Failed    : 0
────────────────────────────────────────────────────────────────
{
  "mode": "batch",
  "total_batches": 39,
  "start_batch": 1,
  "end_batch": 39,
  "successful_batches": 39,
  "failed_batches": 0,
  "batches": [
    {
      "batch_number": 8,
      "files": ["2._a_tale_of_two_cities_au_part1.txt"],
      "success": true,
      "processing_time": 22.58,
      "elapsed_ms": 22551.1,
      "response": {
        "reply": {
          "content": "1 file(s) uploaded successfully",
          "docs": {
            "new_files": ["2._a_tale_of_two_cities_au_part1"],
            "updated_files": [],
            "failed_files": []
          }
        },
        "message": "1 file(s) uploaded successfully",
        "status_code": 200
      }
    }
  ]
}
```

**Resuming after a crash** — set `START_FROM` to last successful batch + 1:
```python
START_FROM = 15   # batches 1–14 already done
END_FROM   = 3800
```

---

### `run_refresh_direct()`

Updates files that **already exist** in the namespace, passing updated content directly as strings.

**Sample output:**
```
────────────────────────────────────────────────────────────
  ▶  REFRESH_DOCUMENTS  (direct mode)
────────────────────────────────────────────────────────────
making request
   ✓  Done
────────────────────────────────────────────────────────────
{
  "reply": {
    "content": "1 file(s) updated successfully",
    "docs": {
      "new_files": [],
      "updated_files": ["test1"],
      "failed_files": []
    }
  },
  "message": "1 file(s) updated successfully",
  "status_code": 200
}
```

---

### `run_refresh_path()`

Same as `run_add_path()` but calls the `refresh_docs` endpoint. Produces identical console output (banner, progress bar, completion summary). The server rejects any file that does not already exist in the namespace.

```
────────────────────────────────────────────────────────────────
  ▶  REFRESH_DOCS
────────────────────────────────────────────────────────────────
   Source files  : 152
   Chunks        : 164
   This run      : 39 batches
   Batch range   : #1 → #39
────────────────────────────────────────────────────────────────
refresh_docs:   0%|          | 0/39 [00:00<?, ?batch/s]
making request
...
```

---

### `run_sync()` Recommended

Classifies every file in the upload folder and automatically routes it — no need to know the server state in advance.

**Sample output:**
```
────────────────────────────────────────────────────────────────
  🔍  CLASSIFYING FILES
────────────────────────────────────────────────────────────────
classify: 100%|████████████████████| 152/152 [00:02<00:00, file/s]

   NEW=12  CHANGED=5  UNCHANGED=135

────────────────────────────────────────────────────────────────
  ▶  ADD_DOCS          (12 new files)
────────────────────────────────────────────────────────────────
...
────────────────────────────────────────────────────────────────
  ▶  REFRESH_DOCS      (5 changed files)
────────────────────────────────────────────────────────────────
...
────────────────────────────────────────────────────────────────
  ■  SYNC COMPLETE
────────────────────────────────────────────────────────────────
   Added     : 12
   Refreshed : 5
   Skipped   : 135
   Failed    : 0
────────────────────────────────────────────────────────────────
```

---

### `run_sync_dry_run()`

Previews the sync plan without touching the server. Run this first to verify what will be uploaded.

**Sample output:**
```
────────────────────────────────────────────────────────────────
  ■  DRY RUN COMPLETE — no files uploaded
────────────────────────────────────────────────────────────────
   NEW=12  REFRESH=5  SKIP=135
────────────────────────────────────────────────────────────────
{
  "mode": "dry_run",
  "plan": { "to_add": 12, "to_refresh": 5, "to_skip": 135, "total": 152 },
  "classifications": [
    { "filename": "report.pdf",  "status": "new",       "endpoint": "add_docs",     "reason": "No chunks on disk — first upload" },
    { "filename": "notes.txt",   "status": "changed",   "endpoint": "refresh_docs", "reason": "Content differs from existing chunks" },
    { "filename": "data.csv",    "status": "unchanged", "endpoint": "skip",         "reason": "Content matches existing chunks — skipping" }
  ]
}
```

---

### `run_match_static(query)` — `search_type="flat"`

Full-corpus fusion. Both semantic search and hybrid filter scan ALL documents independently; results are fused by the proprietary ranking algorithm into a three-tier ranked list.

**Sample output:**
```
──────────────────────────────────────────────────────────────────────
  MATCHING DOCS — flat (all docs)
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": [
      {
        "file_name": "electrolytes-test-report-f_part1",
        "doc_score": 199.95,
        "category": "Very Good Match",
        "chunks": {
          "chunk": [
            "Yash M. Patel Age: 21 Years Sex: Male PID: 555 ...
             Sodium 110.00 Low | Potassium 8.00 High | Chloride 99.00 ..."
          ],
          "similarities": [2.0]
        }
      },
      {
        "file_name": "thyroid-profile-test-repor_part1",
        "doc_score": 199.95,
        "category": "Very Good Match",
        "chunks": {
          "chunk": [
            "Yash M. Patel Age: 21 Years ... TSH 10.10 High 0.40-4.00 mU/L ..."
          ],
          "similarities": [2.0]
        }
      }
    ],
    "docs": {
      "Referred_files": [
        "electrolytes-test-report-f_part1",
        "thyroid-profile-test-repor_part1"
      ],
      "Failed_files": []
    }
  },
  "message": "Data fetched successfully",
  "status_code": 200
}
```

---

### `run_match_filtered(query)` — `search_type="flat_filter"`

Precision search. Only documents that pass the hybrid keyword filter enter the semantic search stage — giving tighter, more focused results.

**Sample output:** Identical structure to `flat` above. The difference is in the candidate pool — semantic scoring only runs over filter-passing documents, so the result set is narrower and more precise.

---

### `run_match_dynamic(query)` / `run_match_dynamic_filtered(query)`

Same as static search but over **temporary files you provide at query time** — nothing needs to be pre-indexed. Content is passed inline via `files_name` + `files_data`.

---

### `run_namespace_details()`

Returns storage and quota info for your namespace(s).

**Sample output:**
```
──────────────────────────────────────────────────────────────────────
  NAMESPACE DETAILS
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "content": [
      {
        "vector_lake_description": "customdataset_",
        "faiss_disk_size_mb": "14.43 mb",
        "index_store_used_mb": "0.0039 mb",
        "index_store_quota_remaining_mb": "24576.00 mb",
        "source_store_files": 166,
        "source_store_used_mb": "30.9062 mb",
        "source_store_quota_remaining_mb": "12257.09 mb"
      },
      {
        "total vector lake": 1,
        "total files": 166,
        "total index_disk_size_mb": "14.43 mb",
        "source_store_quota_used_pct": "0.25%"
      }
    ]
  },
  "message": "Information retrieved successfully",
  "status_code": 200
}
```

---

### `run_docs_info()`

Lists all document names currently stored in the namespace.

**Sample output:**
```
──────────────────────────────────────────────────────────────────────
  DOCS INFORMATION
──────────────────────────────────────────────────────────────────────
{
  "reply": {
    "docs": [
      "001-hide-and-seek-free-chi",
      "002-ginger-the-giraffe-fre",
      "1._the_three_musketeers_au",
      "2._a_tale_of_two_cities_au",
      "2502_19920v2",
      "2505_00851v1",
      ...
    ]
  },
  "message": "Information retrieved successfully",
  "status_code": 200
}
```

---

## 📚 VectorLake Client API Reference

All public methods return a plain `dict` and **never raise** — errors are surfaced as `{"success": False, "error": "...", "message": "..."}`.

### 1. `add_documents`

Uploads **new** documents to the index. The server rejects files already present in the namespace — use `sync_documents()` for mixed folders.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier (registration email). |
| `vector_lake_description` | str | required | Target namespace. |
| `files_name` + `files_data` | List[str] | — | **Direct mode**: names and string contents. Both required together, equal length. |
| `files` | List[str] | `None` | **Batch mode**: filenames to read from `vector_lake_path`. Defaults to all supported files. |
| `start_from_batch` | int | `1` | Resume point for batch mode. |
| `end_batch` | int | `None` | Upper batch bound (inclusive). `None` = process all. |
| `intelligent_segmentation` | bool | `True` | Enables server-side segmentation before embedding. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 2. `refresh_documents`

Updates **existing** documents in the index. Same parameters and modes as `add_documents`. The server rejects files that do not already exist. Use `sync_documents()` for mixed folders.

---

### 3. `sync_documents`

Intelligently syncs a folder by routing each file to the correct endpoint based on MD5 comparison against existing chunks on disk.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Target namespace. |
| `files` | List[str] | `None` | Files to sync. Defaults to all supported files in `vector_lake_path`. |
| `dry_run` | bool | `False` | If `True`, returns classification plan without uploading. |
| `start_from_batch` | int | `1` | Resume point. |
| `end_batch` | int | `None` | Upper batch bound. |
| `intelligent_segmentation` | bool | `True` | Server-side segmentation. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 4. `get_matching_docs`

Retrieves top-matching document chunks using semantic search with optional hybrid filtering.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | str | required | Natural language or WaveQL search query. |
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Target namespace. |
| `pattern` | str | `"static"` | `"static"` for indexed docs; `"dynamic"` for temporary files. |
| `hybrid_filter` | bool | `False` | Enables keyword hybrid filter. **Required when `search_type` is set.** |
| `search_type` | str | `None` | `"flat"` — full corpus fusion (semantic + filter fused over ALL docs). `"flat_filter"` — semantic search over filter-passing docs only. `None` — server default. |
| `top_docs` | int | `10` | Maximum chunks to return. |
| `threshold` | float | `0.2` | Similarity score cutoff. |
| `with_data` | bool | `False` | If `True`, includes raw chunk text in response. |
| `files_name` + `files_data` | List[str] | `None` | Dynamic mode: temporary files to search over. |
| `session_id` | str | `None` | Optional session identifier. |

**Validation rules:**
- `search_type` must be `"flat"` or `"flat_filter"`; anything else returns `InvalidSearchTypeError`.
- `hybrid_filter=True` is required whenever `search_type` is specified.

---

### 5. `health_check`

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Namespace to check. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 6. `get_namespace_details`

Returns storage and quota metadata for one or all namespaces belonging to the user.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | `None` | Scopes response to that namespace if supplied. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 7. `get_docs_information`

Returns document-level metadata within a namespace, optionally filtered by keyword.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Namespace to query. |
| `keyword` | str | `None` | Optional filter keyword. |
| `threshold` | int | `70` | Keyword-match threshold. |
| `session_id` | str | `None` | Optional session identifier. |

---

### 8. `full_corpus_search`

Full-text keyword search across all documents in a namespace.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `user_id` | str | required | User identifier. |
| `vector_lake_description` | str | required | Namespace to search. |
| `keyword` | str | required | Search term. |
| `top_docs` | int | `10` | Maximum results to return. |
| `session_id` | str | `None` | Optional session identifier. |

---

##  Error Handling

All exceptions inherit from `VectorLakeError` and expose `.to_response()`. Since all public methods catch and convert exceptions internally, you only ever need to check the `"error"` key in the returned dict.

| Exception | When raised |
|-----------|------------|
| `ConfigError` | API key missing or config invalid at init time. |
| `ValidationError` | Mismatched `files_name`/`files_data` lengths or other pre-flight failures. |
| `InvalidSearchTypeError` | `search_type` is not `"flat"` or `"flat_filter"`. |
| `UnsupportedFileTypeError` | File extension not in `allowed_extensions`. |
| `FileProcessingError` | File I/O, encoding, or parsing failure. |
| `APIError` | HTTP 4xx/5xx from server. Carries `.status_code` and `.response_text`. |
| `ThrottleError` | HTTP 429 rate limit. Carries `.retry_after`. |

---

##  Structured Logging

The `Logger` class writes three CSV files to `log_dir` (default: `logs/`):

| File | Contents |
|------|----------|
| `performance.csv` | Per-request latency, payload/response KB, HTTP status |
| `api_errors.csv` | Operation, batch number, error message |
| `skipped_files.csv` | Filename and reason when a file is skipped |

---

##  Retry and Backoff

```
  Request attempt 1
      │
      ├── HTTP 429     → wait Retry-After header (or 2^attempt s) → retry
      ├── Timeout      → wait 2^attempt s → retry
      ├── ConnError    → wait 2^attempt s → retry
      ├── HTTP 4xx/5xx → return error dict immediately (no retry)
      └── Success      → return response dict
```

Max retries: 2 (configurable via `VECTOR_LAKE_MAX_RETRIES`).

---

##  Using WaveQL (VQL) Queries

WaveQL enables natural language filtering using brace-based logical groups.

### Syntax at a Glance

```
  ┌──────────────────────────────────────────────────────────────────┐
  │  {(clinical trials) or (observational studies)} {diabetes}       │
  │   └──────────────── group 1 ─────────────────┘  └── group 2 ──┘ │
  │                                                                  │
  │  Groups combine with implicit AND:                               │
  │  → (clinical trials OR observational studies) AND diabetes       │
  └──────────────────────────────────────────────────────────────────┘

  Within a group:
  ┌──────────────────────────────────────────────────────────┐
  │  {A and B}       → both A and B must match               │
  │  {A or B}        → either A or B                         │
  │  {(A B) or C}    → phrase "A B", or term C               │
  │  {A B}           → implicit AND: A and B                 │
  └──────────────────────────────────────────────────────────┘

  ✅ Correct                        ❌ Wrong
  ──────────────────────────────    ─────────────────────────────
  {(machine learning) or            {machine learning or
   (deep learning)}                  deep learning}
                                      → "machine" treated as
                                        a separate term

  {(product manager) or             {product manager or Delhi}
   (data scientist)}                  → ambiguous parse
```

### Three-Tier Result Ranking

```
  ┌─────────────────────────────────────────────────────────────┐
  │             RESULT TIERS (highest → lowest)                 │
  ├──────────┬──────────────────────────────────────────────────┤
  │  Tier 1  │  ✓ Filter match  +  ✓ Semantic match             │
  │          │  Highest confidence — structure & meaning align  │
  ├──────────┼──────────────────────────────────────────────────┤
  │  Tier 2  │  ✓ Filter match  +  ✗ Semantic match             │
  │          │  Structured match, lower semantic relevance      │
  ├──────────┼──────────────────────────────────────────────────┤
  │  Tier 3  │  ✗ Filter match  +  ✓ Semantic match             │
  │          │  Meaning-based fallback, no filter alignment     │
  └──────────┴──────────────────────────────────────────────────┘
```

### Query Examples by Domain

**Healthcare:**
```
Select Top 10 
Where QUERY IS "Need detail about disease state progression of partient id 555
Contains {diabetes} {(clinical trial)} {PID 555}
```

**Recruitment:**
```
Select Top 30 
Where QUERY IS " Need list of resources who have good experience in python , and machine learning"
Contains {Python} {(machine learning)} {Delhi}

```

### Filter Design Best Practices

**DO:** Use 1–2 keywords per brace · Wrap multi-word phrases in parentheses with operators · Keep filters domain-consistent · Trust semantic fallback for edge cases

**DON'T:** Use 5+ word phrases · Mix unrelated domains (`{resume} {clinical trials}`) · Forget parentheses around multi-word phrases with OR/AND · Over-specify filters

---

## 🎯 No Schema Required

```
  Traditional approach                  WaveflowDB approach
  ────────────────────                  ───────────────────
  1. Define JSON schema           vs.   1. Upload raw files
  2. Extract & map every field             (PDF, txt, docx, csv…)
  3. Maintain schema consistency
  4. Update schema for new fields   →   2. Query immediately
  5. Re-index on schema change             with WaveQL
```

| Feature | Traditional | WaveflowDB |
|---------|-------------|------------|
| Data Ingestion | Extract, map, validate | Direct upload |
| Schema Definition | Required upfront | Not required |
| Query Capability | Exact field matching | Semantic + logical filtering |
| New Document Types | Requires schema update | Works immediately |
| Maintenance | High | Low |

---

## 📧 Support

For API or platform support, visit: **https://db.agentanalytics.ai**

---

## 📄 License

Copyright DIBR tech private ltd.
