Metadata-Version: 2.4
Name: multimodal-agent
Version: 0.7.0
Summary: A Python multimodal agent for interacting with Gemini models via text, images, and CLI.
Author-email: Horam <horam.tech@google.com>
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: google-genai>=1.45.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: Pillow>=10.0.0
Provides-Extra: test
Requires-Dist: pytest>=7.4; extra == "test"
Requires-Dist: pytest-mock>=3.12; extra == "test"
Requires-Dist: pytest-cov>=4.1; extra == "test"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-mock>=3.12; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: flake8>=7.0; extra == "dev"
Requires-Dist: isort>=5.13; extra == "dev"
Requires-Dist: mkdocs>=1.6; extra == "dev"
Requires-Dist: mkdocs-material>=9.5; extra == "dev"
Requires-Dist: mkdocstrings[python]; extra == "dev"
Dynamic: license-file

# **Multimodal-Agent**

*A lightweight, production-ready multimodal wrapper for Google Gemini with RAG, image input, JSON mode, project learning, session memory, and a clean CLI & server.*

---

## Features

### Core LLM Capabilities

* Unified agent for **text, image, and chat** interactions
* Clean  **CLI** : `agent ask`, `agent image`, `agent chat`, `agent history`, `agent learn-project`
* Supports  **Gemini 2.5-flash** ,  **1.5-flash** , and any future model (configurable)
* Automatic **retry logic with exponential backoff**
* Full offline mode support (`FAKE_RESPONSE`) when no API key is available
* Detailed  **usage logging** : prompt, response, and total token counts

### **RAG + Memory**

* Local SQLite **RAGStore** (no cloud dependency)
* Automatic memory saving of past chats
* Project learning: let the agent read source code & architecture
* Project introspection commands: `learn-project`, `show-project`, `inspect-project`

### **Configuration System**

* User config stored at: `~/.multimodal_agent/config.yaml`
* Configure models individually:
  * `chat_model`
  * `image_model`
  * `embedding_model`
* New CLI commands:
  * `agent config set-model <model>`
  * `agent config set-image-model <model>`
  * `agent config set-embed-model <model>`
  * `agent config set-key <API_KEY>`

### **Developer Experience**

* pytest fixtures for offline/fake mode
* High test coverage rate
* Type-safe `AgentResponse`
* Extensible architecture
* Easy to embed into apps or scripts

---

## Installation

```bash
pip install multimodal-agent
```

Or local:

```bash
pip install -e .
```

---

# Configuration

### Show current configuration:

```bash
agent config show
```

### Set API key:

```bash
agent config set-key YOUR_KEY
```

### Set chat model:

```bash
agent config set-model gemini-2.5-flash
```

### Set image model:

```bash
agent config set-image-model gemini-1.5-flash
```

### Set embedding model:

```bash
agent config set-embed-model text-embedding-004
```

Your config file after updates:

```bash
local_learning: true
chat_model: gemini-2.5-flash
image_model: gemini-2.0-flash
embedding_model: text-embedding-004
api_key: YOUR_KEY
```

---

## Quick Start

### **Text Question**

```bash
agent ask "What is the capital of France?"
```

### **Disable RAG**

```bash
agent ask "What is the capital of France?" --no-rag
```

### **JSON mode**

```bash
agent ask "give me json" --json
```

### **Image + Text**

```bash
agent image test.jpg "describe this"
```

### **Chat (with persistent memory)**

```bash
agent chat
```

---

### History / Memory

Your memory DB lives at:

```bash
~/.multimodal_agent/memory.db
```

Show memory:

```bash
agent history show
```

Clear memory:

```bash
agent history clear
```

Summarize memory:

```bash
agent history summary
```

---

## Learning a Project

Let the agent scan and store a project summary:

```bash
agent learn-project my_app/
```

List learned projects:

```bash
agent list-projects
```

Show a specific project:

```bash
agent show-project project:my_app
```

Inspect project without saving:

```bash
agent inspect-project my_app/
```

---

### Python API Example

```python
from multimodal_agent.core.agent_core import MultiModalAgent

agent = MultiModalAgent()

resp = agent.ask("Explain quantum computing")
print(resp.text)
print(resp.usage)

```

Image example:

```python
from multimodal_agent.utils import load_image_as_part

img = load_image_as_part("cat.jpg")
resp = agent.ask_with_image("describe this", img)
print(resp.text)
```

---

### Server Mode

Start:

```bash
agent server
```

Runs at:

```
http://127.0.0.1:8000
```

## API Reference (v0.6.0)

## **POST /ask**

```bash
curl -X POST http://127.0.0.1:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"prompt": "hello"}'
```

Response:

```json
{
  "text": "hello",
  "data": null,
  "usage": { "prompt_tokens": 44, "response_tokens": 3, "total_tokens": 553 }
}
```

## **POST /ask_with_image**

```bash
curl -X POST http://127.0.0.1:8000/ask_with_image \
  -F "file=@test.jpg" \
  -F "prompt=describe this"
```

### v0.6.0 Better Error Handling

Failures now return:

```json
{
  "text": "Image processing failed: 429 RESOURCE_EXHAUSTED ...",
  "data": null,
  "usage": {},
  "error": true
}
```

Never returns `text: null`.

---

## **POST /generate**

```bash
curl -X POST http://127.0.0.1:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "give me json", "json": true}'
```

---

## **POST /memory/search**

```bash
curl -X POST http://127.0.0.1:8000/memory/search \
  -H "Content-Type: application/json" \
  -d '{"query": "hello"}'
```

Response:

```json
{
  "results": [
    [0.98, { "id": 1, "content": "hello", "role": "user" }]
  ]
}
```

---

## **POST /learn/project**

Returns a structured project profile:

```json
{
  "status": "ok",
  "project_id": "project:rope_simulation_using_flutter",
  "profile": {
    "package_name": "rope_simulation_using_flutter",
    "architecture": {
      "patterns": ["feature_first"],
      "state_management": []
    },
    "dart_files_count": 3,
    "widget_files_count": 2
  }
}
```

---

## Architecture Overview

```bash
multimodal_agent/
    core/          # Main agent logic
    rag/           # SQLite vector store
    cli/           # CLI commands (`agent`)
    server/        # FastAPI server implementation
    utils/         # helpers
```

### Memory schema:

```bash
sessions      # chat sessions
chunks        # tokenized fragments
embeddings    # vector embeddings
projects      # project profiles (v0.6.0)
```

---

## **Formatting Engine (v0.4.0+)**

* Detects JSON, XML, HTML, code, python, kotlin, dart, js, swift …
* Pretty-prints output
* Auto-wraps in fenced code blocks
* Optional in `agent.ask(formatted=True)`

---

## Running Tests

```bash
make test
make coverage
```

This includes:

* RAG tests
* CLI tests
* JSON mode tests
* Fake mode (offline)
* Config isolation
* SQLite operations

---

## Roadmap

### **v0.8.0**

* Streaming responses
* Conversations with images
* Project-diff memory updates

### **v1.0**

* Stable API
* Plugin ecosystem
* Multi-language project analyzers

---

# License

MIT License.
