Metadata-Version: 2.4
Name: openmind-core
Version: 0.0.4
Summary: Local-first AI memory engine for user-approved folders.
Project-URL: Homepage, https://github.com/codewithbro95/openmind
Project-URL: Repository, https://github.com/codewithbro95/openmind
Project-URL: Issues, https://github.com/codewithbro95/openmind/issues
Project-URL: Changelog, https://github.com/codewithbro95/openmind/blob/main/CHANGELOG.md
Author: OpenMind contributors
License: MIT
License-File: LICENSE
Keywords: ai,cli,lancedb,lm-studio,local-first,memory,rag,search
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Utilities
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: lancedb>=0.13.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: pillow>=12.3.0
Requires-Dist: pydantic>=2.7.0
Requires-Dist: pypdf>=4.2.0
Requires-Dist: pypdfium2>=5.11.0
Requires-Dist: python-docx>=1.1.0
Requires-Dist: rapidocr-onnxruntime>=1.4.4
Requires-Dist: rich>=13.7.0
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: typer>=0.12.0
Provides-Extra: dev
Requires-Dist: pytest>=8.2.0; extra == 'dev'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/openmind-banner.png" alt="OpenMind banner" width="100%">
</p>

# OpenMind Core

OpenMind is a local AI memory engine for your computer.

It indexes folders you explicitly approve, stores searchable memory locally, and lets you search or ask questions across your own files with sources attached.

OpenMind is not a chatbot, desktop UI, browser extension, cloud sync service, or agent that controls your machine. It focuses on three core jobs:

```text
Index local files -> Search local memory -> Ask source-grounded questions
```

## Why am i building this?

I am building this because I have a lot of files on my computer, and sometimes I genuinely get lost. I forget a file I downloaded months ago, or something I saved years ago. sometims i just need an order id from a receipt i got last week, a flight number from a pdf, or a detail buried somewhere in my messy downloads folder or even across my system. I do not want to upload all of that to another app just to find it again, no. or even give access to my data to big tech companies. I want a local AI memory that quietly understands the folders I already have existing on my system, works in the background, and helps me ask my own computer what it already has, that's it

## What OpenMind Does

OpenMind Core is a local-first CLI for indexing, searching, and asking questions over user-approved folders.

- Local app storage under `~/.openmind`.
- User-approved folder sources.
- File extraction for common text, PDF, DOCX, CSV, Markdown, HTML, and image files.
- LanceDB vector storage.
- SQLite source, file, and indexing job records.
- LM Studio as the user-facing local AI provider.
- Background indexing with live progress.
- Streaming answers by default.
- Interactive ask sessions with temporary conversation memory.
- Source-grounded answers.
- Developer log inspection.

OpenMind intentionally avoids:

- desktop UI
- browser extension
- cloud sync
- file automation
- plugin marketplace
- deleting, moving, or modifying user files

See [FEATURES.md](FEATURES.md) for the complete shipped feature list and roadmap. See [CHANGELOG.md](CHANGELOG.md) for release notes.

## Requirements

- Python 3.11+
- `uv`
- LM Studio for local chat, embedding, and vision models
- macOS, Linux, or another Python-supported environment

OpenMind Core `0.0.4` uses LM Studio as its only user-facing provider. The older Sentence Transformers provider remains only as a development and test fallback.

## Install

Install the CLI with `uv`:

```bash
uv tool install openmind-core
```

Start setup:

```bash
openmind setup
```

You can also install it with pipx:

```bash
pipx install openmind-core
```

The PyPI package is named `openmind-core`; the command it installs is `openmind`.

## Development
For local development, clone the project and install it into your Python environment.

If you already have a conda environment named `openmind` (you can name it whatever you want):

```bash
cd openmind-core
conda activate openmind
uv pip install -e ".[dev]"
pytest
```

`uv pip install` detects the activated conda environment and installs the packages into it, while still using uv's fast resolver and installer.

To let `uv` manage the environment:

```bash
cd openmind-core
uv sync --all-extras
uv run pytest
```

Useful dependency commands:

```bash
uv lock
uv sync --all-extras
uv pip install -e ".[dev]"
```

## Quick Start

Start the LM Studio local server first. In LM Studio, open the Developer tab and start the server, or run:

```bash
lms server start
```

Run first-time setup:

```bash
openmind setup
```

Setup:

1. Initialize `~/.openmind`.
2. Check that LM Studio is reachable.
3. Let you choose LM Studio as the provider.
4. List available chat, embedding, and image description models.
5. Load the selected models.
6. Ask which folders to index.
7. Start background indexing.

Watch indexing progress:

```bash
openmind index status
```

The live status table includes an `Already indexed` count for unchanged files that were indexed before and are still accessible.

Search your local memory:

```bash
openmind search "holiday plan"
```

Ask a question with sources:

```bash
openmind ask "What documents do I have about the cabin trip?"
```

Start an interactive ask session:

```bash
openmind ask
```

## CLI Reference

Start here:

```bash
openmind setup
```

Lower-level initialization:

```bash
openmind init
openmind status
openmind flush
openmind flush --dry-run
openmind flush --yes
openmind flush --yes --include-sources
openmind uninstall
openmind uninstall --dry-run
openmind uninstall --yes
openmind uninstall --yes --package
```

Source management:

```bash
openmind source add ~/Documents
openmind source list
openmind source remove <source_id>
```

If a folder was already added, OpenMind tells you it is already registered and reports indexed files that are already accessible.

Indexing:

```bash
openmind index
openmind index start
openmind index status
openmind index status --once
openmind index pause
openmind index resume
openmind index stop
```

If an unchanged file was indexed before, OpenMind reports it as already indexed and keeps it available for search and ask.

OpenMind uses file path, size, modified time, and content hash to avoid unnecessary work. Unchanged files are not extracted, embedded, or stored again. If a file's metadata changes, OpenMind checks the content hash and only re-indexes when the content actually changed.

Search:

```bash
openmind search "holiday plan"
openmind search "OAuth redirect issue" --limit 10
```

Ask:

```bash
openmind ask "What do my files say about the cabin trip?"
openmind ask "What do my files say about the cabin trip?" --no-stream
openmind ask "What do my files say about the cabin trip?" --show-thinking
openmind ask "What do my files say about the cabin trip?" --limit 8
openmind ask
```

Interactive ask commands:

```text
/clear  reset the current session memory
/exit   leave the chat
/quit   leave the chat
```

LM Studio provider commands:

```bash
openmind provider status
openmind models list
openmind models load
openmind models load <model_key>
openmind models update
openmind models update --no-load
```

Developer logs:

```bash
openmind dev logs
openmind dev logs --no-follow --lines 40
openmind dev logs --log all
openmind dev logs --log index
openmind dev logs --lm-studio
```

## LM Studio Integration

OpenMind talks to LM Studio at:

```text
http://localhost:1234
```

It uses LM Studio's native REST API for model setup:

```text
GET  /api/v1/models
POST /api/v1/models/load
```

It uses OpenAI-compatible endpoints for inference:

```text
POST /v1/chat/completions
POST /v1/responses
POST /v1/embeddings
```

OpenMind stores separate model choices because chat, embeddings, and image descriptions are different jobs:

```toml
[provider]
name = "lmstudio"
base_url = "http://localhost:1234"
api_token_env = "LM_API_TOKEN"

[models]
chat_model = "selected-chat-model-key"
embedding_model = "selected-embedding-model-key"

[indexing]
auto_start_after_setup = true
background = true

[extraction.images]
enabled = true
model = "selected-vision-model-key"
```

Change saved models with:

```bash
openmind models update
```

The command fetches the latest LM Studio model list, lets you choose chat, embedding, and image description models, saves the new config, and loads the selected models by default.

When loading or updating models, OpenMind checks LM Studio first and skips models that are already loaded.

If LM Studio is not running, OpenMind exits with a clear message instead of a Python traceback.

## Architecture Choices

OpenMind keeps the architecture intentionally simple. Each technology has one simple job.

### High-Level Overview

OpenMind is the memory engine and CLI. It does not use LM Studio's chat interface, or any other provider's chat UI. It calls a local model server endpoint to reach downloaded models.

The current local model server is LM Studio. The provider layer is designed for additional local or OpenAI-compatible servers.

```mermaid
flowchart TD
    User["User"] --> CLI["OpenMind CLI"]

    CLI --> Setup["Setup / Config"]
    CLI --> Sources["Source Manager"]
    CLI --> Indexing["Indexing Engine"]
    CLI --> Search["Search"]
    CLI --> Ask["Ask"]

    Setup --> Config["~/.openmind/config.toml"]
    Setup --> ModelServer["Local Model Server<br/>(LM Studio, other providers later)"]

    Sources --> SQLite["SQLite<br/>sources, files, jobs, status"]

    Indexing --> Scanner["File Scanner<br/>user-approved folders only"]
    Scanner --> Extractors["Extractors<br/>txt, md, pdf, docx, csv, html"]
    Extractors --> Normalizer["Normalize Text"]
    Normalizer --> Chunker["Chunk Text"]
    Chunker --> Embeddings["Embedding Provider<br/>calls local embedding endpoint"]
    Embeddings --> ModelServer
    Embeddings --> LanceDB["LanceDB<br/>vectors + chunks"]

    Search --> QueryEmbed["Embed Query"]
    QueryEmbed --> ModelServer
    QueryEmbed --> LanceDB
    LanceDB --> Results["Relevant Chunks<br/>path, score, snippet"]

    Ask --> Search
    Results --> Context["Build Context<br/>with sources"]
    Context --> AnswerModel["Answer Provider<br/>calls local chat/completion endpoint"]
    AnswerModel --> ModelServer
    AnswerModel --> Answer["Answer with Sources"]

    Indexing --> Jobs["Background Index Job"]
    Jobs --> SQLite
    CLI --> Logs["Dev Logs"]
    Logs --> LogFiles["~/.openmind/logs"]
```

### SQLite

SQLite is used for **project state and metadata**, not the AI memory itself.

SQLite stores:

- sources and folders the user added
- file paths and file hashes
- indexing status
- indexing progress
- config and local state
- failed files or skipped files
- background job info

Why SQLite:

- it is local and embedded
- it needs no separate server
- it is reliable for small structured records
- it makes indexing progress easy to inspect and resume

### LanceDB

LanceDB is used for **searchable AI memory**.

LanceDB stores:

- extracted text chunks
- embeddings and vectors
- chunk metadata
- source paths for search results and answers

Why LanceDB:

- it runs locally from a directory path
- it avoids a separate vector database server
- it is designed for vector search
- it keeps OpenMind's memory layer portable

Simple way to think about it:

> **SQLite keeps track of what OpenMind is doing. LanceDB stores what OpenMind knows.**

### Model Provider

OpenMind uses a model provider abstraction for embeddings and answers.

In `0.0.4`, the only implemented user-facing provider is LM Studio. OpenMind talks to LM Studio's local server endpoint; it does not use the LM Studio chat interface.

OpenMind uses the provider endpoint for:

- embedding local file chunks
- embedding search queries
- generating source-grounded answers
- streaming answer tokens in ask mode
- generating image descriptions for image indexing

Why LM Studio first:

- it runs local models on the user's machine
- it exposes a local API server
- it supports OpenAI-compatible chat, embedding, and multimodal endpoints
- it lets OpenMind stay local-first without owning model runtime complexity

Future providers can fit behind the same layer, such as Ollama, llama.cpp, or another OpenAI-compatible local endpoint.

### Typer and Rich

Typer powers the CLI. Rich powers readable terminal output.

Why they are used:

- Typer keeps commands small and type-friendly
- Rich makes tables, progress views, and errors easier to read
- the CLI stays usable before any desktop or web UI exists

### uv

uv is used for dependency management and development setup.

Why uv:

- fast installs and dependency resolution
- works with an existing conda environment
- supports reproducible lockfiles
- keeps contributor setup simple

## Supported Files

OpenMind indexes:

```text
.txt
.md
.pdf
.docx
.csv
.html
.png
.jpg
.jpeg
.webp
.bmp
.tif
.tiff
```

OpenMind is document-first by default. It does not index source code, JSON config files, package metadata, app asset catalogs, or other low-level project internals unless a dedicated indexing mode is enabled. High-level project documents such as `README.md`, Markdown notes, PDFs, DOCX files, CSVs, and HTML docs can still be indexed.

PDF extraction first uses the normal embedded text layer. If a PDF looks scanned or the extracted text is too sparse, OpenMind automatically tries local OCR with RapidOCR + ONNX Runtime and then continues the normal indexing pipeline.

It ignores noisy folders such as:

```text
.git
node_modules
venv
.venv
.env
__pycache__
dist
build
.cache
target
coverage
Assets.xcassets
hidden folders
```

## Image Indexing

OpenMind can index standalone images through a local vision model served by LM Studio. The recommended first model is:

```text
ggml-org/SmolVLM-500M-Instruct-GGUF
```

Image indexing keeps the original image file on disk and does not store raw image bytes in LanceDB.

For supported image files, OpenMind stores:

- file path
- searchable file and image metadata
- generated image description
- OCR text when available
- text embedding of the combined description and OCR text

That means users can search and ask questions about screenshots, photos, scanned image files, labels, UI errors, receipts, and other image-like local files without copying the image itself into the vector database.

Image metadata includes safe, JSON-friendly fields such as dimensions, format, mode, file size, EXIF tags, and text-based image info. Binary metadata fields are summarized by size instead of being stored as raw bytes.

Image config lives in `~/.openmind/config.toml`:

```toml
[extraction.images]
enabled = true
model = "ggml-org/SmolVLM-500M-Instruct-GGUF"
ocr_enabled = true
max_new_tokens = 220
```

During setup or `openmind models update`, OpenMind asks for an image description model separately from chat and embedding models. If no vision model is available, image indexing can be disabled while normal document indexing continues.

## OCR Fallback

OCR is automatic for weak or scanned PDFs. No CLI flag is needed.

OpenMind uses RapidOCR with ONNX Runtime as the default OCR backend. It renders PDF pages locally with `pypdfium2`, runs OCR locally, and continues the same normalize/chunk/embed/store pipeline.

These Python OCR dependencies are installed by the normal project install:

```bash
uv pip install -e ".[dev]"
```

OCRmyPDF is still supported as an optional backend for users who prefer it. That mode requires OCRmyPDF, Tesseract, and Ghostscript installed separately:

```bash
brew install ocrmypdf tesseract ghostscript
```

OCR config lives in `~/.openmind/config.toml`:

```toml
[extraction.ocr]
enabled = true
backend = "rapidocr"
min_text_chars_per_page = 80
```

If OCR dependencies are missing or an optional OCR backend is unavailable, OpenMind does not crash the indexing run. It records a clear extraction error for that file and continues with the rest of the source.

## Search Mode

Search does not require a chat model. It embeds the query with the selected LM Studio embedding model, searches LanceDB, and returns paths, scores, and snippets.

Example:

```bash
openmind search "cabin trip checklist"
```

Output is shaped like:

```text
1. ~/Documents/Holiday/checklist.md
   Score: 0.91
   Snippet: The packing checklist includes...
```

Answer quality depends on retrieval quality. OpenMind treats search as the foundation.

## Ask Mode

Ask is search plus an answer model:

```text
question
  -> retrieve relevant chunks
  -> build grounded context
  -> stream answer from LM Studio
  -> show sources
```

Answers stream by default:

```bash
openmind ask "What do my files say about the cabin trip?"
```

Disable streaming when needed:

```bash
openmind ask "What do my files say about the cabin trip?" --no-stream
```

Show provider-returned thinking or reasoning when the selected LM Studio model exposes it:

```bash
openmind ask "What do my files say about the cabin trip?" --show-thinking
```

If the model does not return explicit thinking or reasoning, OpenMind says so and still returns the answer with sources.

Bare `openmind ask` starts a chat-like session:

```bash
openmind ask
```

Session history is held in memory while the process is open, so follow-up questions can refer to earlier turns. The session is discarded when you exit.

## Background Indexing

Start indexing in the background:

```bash
openmind index start
```

Watch a live table:

```bash
openmind index status
```

Print status once:

```bash
openmind index status --once
```

Pause, resume, or stop:

```bash
openmind index pause
openmind index resume
openmind index stop
```

Indexing has two phases:

1. Discovery: scan enabled sources and count supported files.
2. Indexing: extract, chunk, embed, and store chunks while updating SQLite progress.

The live table shows:

- Job id
- State
- Files discovered
- Files processed
- Files indexed
- Files skipped
- Files failed
- Chunks created
- Progress percentage
- Current file

Pause and stop take effect after the current file finishes. If a file is already inside a slow extraction or embedding request, the worker checks the requested state before moving to the next file.

## Logs

OpenMind writes structured logs to:

```text
~/.openmind/logs/openmind.log
```

Index worker logs are written to:

```text
~/.openmind/logs/index-<job-id>.log
```

Watch logs:

```bash
openmind dev logs
```

Show recent logs once:

```bash
openmind dev logs --no-follow --lines 40
```

Watch all OpenMind logs:

```bash
openmind dev logs --log all
```

Watch only index worker logs:

```bash
openmind dev logs --log index
```

Watch LM Studio logs through its CLI:

```bash
openmind dev logs --lm-studio
```

That command runs:

```bash
lms log stream
```

## Local Storage

OpenMind stores app data under `~/.openmind` by default:

```text
~/.openmind/
├── config.toml
├── openmind.sqlite
├── lancedb/
└── logs/
```

For development and tests, use a separate home:

```bash
OPENMIND_HOME=/tmp/openmind-dev openmind status
```

Reset indexed memory without uninstalling:

```bash
openmind flush
```

This clears OpenMind's indexed memory and indexing state, including SQLite file records, index jobs, LanceDB vectors/chunks, and log files. It keeps `config.toml` and saved source folders by default, so you can run `openmind index start` again from a clean memory state. To also clear saved source folder records:

```bash
openmind flush --yes --include-sources
```

Flush never deletes the actual files or folders you indexed.

Remove OpenMind-owned local data:

```bash
openmind uninstall
```

This deletes the OpenMind app home, including `config.toml`, `openmind.sqlite`, `lancedb/`, and `logs/`. It does not delete user source folders, LM Studio, or downloaded models.

To remove the installed package from the current Python environment in the same command:

```bash
openmind uninstall --yes --package
```

## Test Data

This repo includes a small `data/` folder with notes, Markdown, JSON, CSV, HTML, JavaScript, a sample PDF, and images. Supported document-first formats, PDFs, and supported image files can be indexed.

Try it:

```bash
openmind source add ./data
openmind index start
openmind index status
openmind search "holiday plan"
```

## Project Structure

```text
openmind/
├── cli/
├── core/
├── sources/
├── extractors/
├── ingestion/
├── embeddings/
├── storage/
├── retrieval/
├── llm/
└── providers/
```

The design is deliberately boring inside: each stage has a small job, and the provider layer is replaceable without rewriting ingestion, storage, or retrieval.

## Development

Install development dependencies:

```bash
uv pip install -e ".[dev]"
```

Run tests:

```bash
pytest
```

Or with uv:

```bash
uv run pytest
```

Keep docs in sync when behavior changes:

- Update [FEATURES.md](FEATURES.md) when a feature lands.
- Update [CHANGELOG.md](CHANGELOG.md) for user-facing release notes.
- Update [TECHNICAL_SPEC.md](TECHNICAL_SPEC.md) when architecture, schema, or interfaces change.
- Update this README when normal user workflow changes.

## Roadmap

Near-term work:

- Better indexing error inspection.
- Failed-file retry commands.
- Rebuild index command.
- Source enable and disable.
- Hybrid keyword plus vector search.
- Better snippets and citations.
- Better OCR and metadata extraction for screenshots, images, and scanned PDFs.
- Persistent chat sessions.
- Local API for UI clients.
- Additional providers after LM Studio is solid.

The full roadmap lives in [FEATURES.md](FEATURES.md).

## Contributing

OpenMind is early and intentionally small. Good contributions make the core more trustworthy without adding premature surface area.

Start with [CONTRIBUTING.md](CONTRIBUTING.md).

## License

MIT. See [LICENSE](LICENSE).
