Metadata-Version: 2.4
Name: napari-chat-assistant
Version: 1.4.3
Summary: Local chat assistant scaffold for napari image-analysis workflows
Author: Wulin Teo
License-Expression: MIT
Project-URL: Homepage, https://github.com/wulinteousa2-hash/napari-chat-assistant
Project-URL: Repository, https://github.com/wulinteousa2-hash/napari-chat-assistant
Project-URL: Issues, https://github.com/wulinteousa2-hash/napari-chat-assistant/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: napari
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Image Processing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: napari
Requires-Dist: numpy
Requires-Dist: qtpy
Requires-Dist: scipy
Requires-Dist: scikit-image
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Dynamic: license-file

# napari-chat-assistant

[![License MIT](https://img.shields.io/pypi/l/napari-chat-assistant.svg?color=green)](https://github.com/wulinteousa2-hash/napari-chat-assistant/raw/main/LICENSE)
[![PyPI](https://img.shields.io/pypi/v/napari-chat-assistant.svg?color=green)](https://pypi.org/project/napari-chat-assistant)
[![Python Version](https://img.shields.io/pypi/pyversions/napari-chat-assistant.svg?color=green)](https://python.org)
[![napari hub](https://img.shields.io/endpoint?url=https://api.napari-hub.org/shields/napari-chat-assistant)](https://napari-hub.org/plugins/napari-chat-assistant)

---

Local Ollama-powered assistant for napari image-analysis workflows.

`napari-chat-assistant` adds a dock widget inside napari that understands the active viewer session, runs built-in image-analysis actions, and generates executable napari Python code when a request goes beyond the current toolset.

It is designed for local interactive work, repeatable workflows, and gradual automation rather than cloud chat or fully opaque “one-click AI”.

The current direction is a deterministic, layer-aware assistant: the plugin profiles loaded napari layers first, then uses that structured context to guide tool choice and generated code.

## What's New In 1.4.3

- added built-in image grid view for side-by-side comparison of open image layers
- added a built-in action to turn image grid view off and restore hidden non-image layers
- added built-in plugin UI help so users can ask what controls do and get short usage tips
- expanded help coverage for Library, Prompt, code actions, model controls, telemetry, diagnostics, status, and prompt tips
- kept experimental SAM2 under `Advanced` rather than the default toolbar
- kept the clearer `Load`, `Unload`, `Test`, `Setup` model flow and improved status wording

For older release history, see [CHANGELOG.md](CHANGELOG.md).

## Overview

Current capabilities include:
- connect to a local Ollama server
- discover and unload local models from the plugin UI
- inspect the selected layer or named layers with structured layer summaries
- profile loaded layers with deterministic semantic and workflow-aware metadata
- run built-in, registry-backed image-analysis tools from natural-language requests
- support common microscopy workflows such as enhancement, thresholding, cleanup, measurement, projection, and cropping
- support ROI-aware inspection and grayscale value extraction from `Labels` and `Shapes` layers
- automate batch actions across multiple layers when the request applies to all compatible inputs
- generate napari Python code only when no built-in tool is a better fit
- review, copy, validate, and run assistant-generated code from the plugin UI
- paste and run your own Python directly from the Prompt box with `Run My Code`, without opening QtConsole
- reuse built-in demo packs for EM-style grayscale data, fluorescent RGB cell data, SNR sweeps, and messy mask cleanup tests
- save, pin, tag, rename, and reuse prompts through the local Library
- save, tag, rename, and run reusable code through the Library `Code` tab
- keep built-in examples alongside recent and saved items in the Library
- delete selected built-in, recent, saved, or code items from the Library and clear unpinned recent history while keeping saved and pinned items
- keep bounded session memory from approved prior turns
- reject the last assistant outcome from session memory with a thumbs-down control
- optionally configure and run experimental SAM2 segmentation from `Advanced`
- optionally open ND2 conversion, spectral viewer, and spectral analysis widgets from `napari-nd2-spectral-ome-zarr`

The current default model is:
- `nemotron-cascade-2:30b`

## Why This Plugin

Most chat interfaces are detached from the actual napari session. This plugin keeps the assistant inside the viewer and grounds its behavior in the live state of the current napari workspace:
- loaded layers
- the selected layer
- layer shape, dtype, and transforms
- deterministic semantic layer profiling
- labels and mask statistics
- built-in local tool execution
- local Python code generation when needed
- bounded session memory

The goal is not a generic chatbot inside napari. The goal is a layer-aware assistant that can grow into a practical microscopy workflow workbench.

### Local-first by design

The assistant runs on local open-weight models through Ollama:

- no API key required
- no internet dependency
- no cloud services
- no data leaves your workstation

This makes it a better fit for research workflows where users want interactive help, repeatable analysis steps, reusable demo and prompt packs, and direct control over both data and models.

## Current Features

### Session-aware tools

The assistant currently supports built-in tools for:
- listing all layers
- inspecting the selected layer
- inspecting a specific named layer
- CLAHE contrast enhancement for grayscale 2D and 3D images
- batch CLAHE across multiple image layers
- Gaussian denoising for grayscale image layers
- threshold preview
- threshold apply
- batch threshold preview and apply
- mask measurement
- batch mask measurement
- mask cleanup operations such as hole filling, small-object removal, and largest-component selection
- connected-component labeling for binary masks
- per-object measurement table summaries for labels layers
- max-intensity projection for 3D grayscale images
- cropping one layer to the bounding box of another layer
- ROI inspection and grayscale value extraction from labels or shapes regions
- registry-backed tool execution as the foundation for future workflow and pipeline expansion

Layer inspection is now backed by a deterministic profile object that includes:
- `semantic_type`
- `confidence`
- `axes_detected`
- `source_kind`
- metadata flags such as multiscale, lazy/chunked, channel metadata, and wavelength metadata
- recommended and discouraged operation classes
- evidence buckets for debugging and future adapter work

Supported mask operations:
- `dilate`
- `erode`
- `open`
- `close`
- `fill_holes`
- `remove_small`
- `keep_largest`

Additional built-in workflow tools currently exposed through chat include:
- `gaussian_denoise`
- `remove_small_objects`
- `fill_mask_holes`
- `keep_largest_component`
- `label_connected_components`
- `measure_labels_table`
- `project_max_intensity`
- `crop_to_layer_bbox`
- `inspect_roi_context`
- `extract_roi_values`

### Code generation workflows

When a request is not covered by a built-in tool, the assistant can return napari Python code instead of guessing or forcing the wrong tool.

Generated code can be:
- copied to the clipboard
- reviewed and executed from the plugin

You can also paste your own Python directly into the Prompt box and run it from the plugin with `Run My Code`, without switching to QtConsole.

Use assistant-generated code when you want a reusable script or need custom logic beyond the current built-in tools.

Use `Run My Code` when you already have Python you want to test quickly inside the current napari session.

### Optional ND2 and spectral integration

If `napari-nd2-spectral-ome-zarr` is installed, the assistant can open:
- the ND2-to-OME-Zarr export widget
- the Spectral Viewer widget
- the Spectral Analysis widget

This lets chat act as an entry point for Nikon ND2 conversion and spectral workflows without rebuilding those UIs inside this plugin.

Install links:
- GitHub: `https://github.com/wulinteousa2-hash/napari-nd2-spectral-ome-zarr`
- napari Hub: `https://napari-hub.org/plugins/napari-nd2-spectral-ome-zarr.html`

### Experimental SAM2 integration

Version `1.4.3` includes an experimental SAM2 path for users who want box-prompt or point-prompt segmentation inside napari without making SAM2 part of the default assistant workflow.

Behavior:
- SAM2 is accessed from `Advanced`, not from the main toolbar
- `SAM2 Setup` is always available from `Advanced`
- `SAM2 Live` stays disabled until the backend is configured and passes readiness checks
- the rest of the assistant remains usable even if SAM2 is not configured

Current setup expects:
- a working Python environment that already includes the dependencies required by your SAM2 wrapper
- an external SAM2 project path
- a wrapper module exposed as `sam2_wrapper.py` or `sam2_wrapper/__init__.py`
- `segment_image_from_box(...)` for box workflows
- `segment_image_from_points(...)` for point workflows if you want live point prompting
- a valid checkpoint path
- a valid config path

Typical setup flow:
1. Start napari from the environment that contains your SAM2 dependencies.
2. Open `Plugins -> Chat Assistant`.
3. Open `Advanced -> SAM2 Setup`.
4. Enter the SAM2 project path, checkpoint path, config path, and device.
5. Click `Test`.
6. Save the settings.
7. Open `Advanced -> SAM2 Live` when the backend reports ready.

### Selective Session Memory

The assistant now includes bounded session memory with three states:
- `provisional`
- `approved`
- `rejected`

Behavior:
- new assistant outcomes start as provisional
- successful follow-up actions can promote them to approved
- only approved items are sent back to the model as `session_memory`
- current viewer context and current layer profiles always override memory
- `Thumbs Down Last Answer` rejects the most recent memory candidate for the current session

This is intentionally not full transcript memory. The model is still grounded primarily in the current napari viewer state.

### Library

The assistant includes a persistent Library for repeatable workflows and reusable code:
- built-in starter prompts
- built-in demo packs and reusable code examples in the `Code` tab
- recent prompts captured automatically
- saved prompts for reusable tasks
- pinned prompts for high-frequency workflows
- recent and saved code snippets in a separate `Code` tab

Interaction:
- single click loads a prompt or code snippet into the editor
- double click sends a prompt directly or runs a code snippet
- right click can rename or edit tags for saved and recent prompt/code items
- multi-select supports Shift/Ctrl selection for batch actions
- `Delete` can remove saved prompts, recent prompts, code snippets, or hide built-in prompts
- `Clear` removes unpinned recent prompt/code items while keeping saved and pinned items

Logic:
- `saved` means a user-managed prompt you want to keep as your own reusable entry
- `pinned` means keep this prompt surfaced at the top of the library
- a prompt can be pinned without being saved
- built-in prompts are shipped examples; deleting them hides them from the current local library view
- built-in code entries include demo packs and starter `Run My Code` examples
- built-in code entries remain visible even if the same snippet also appears in recent history
- code snippets can be tagged and renamed so they are easier to reference later in workflows

This is designed for users who want repeatable automation without committing everything to full scripting.

## Requirements
- Python 3.9+
- napari
- Ollama installed locally and running on the same machine
- at least one local Ollama model such as `nemotron-cascade-2:30b`

Core Python dependencies used by the plugin are installed with the package itself.

Optional:
- `napari-nd2-spectral-ome-zarr` for ND2 export, spectral viewer, and spectral analysis integration
- external SAM2 project, weights, and config if you want the experimental SAM2 workflow

Notes:
- The plugin does not bundle the Ollama server or model weights.
- Model memory requirements vary substantially by model tag.
- Larger local models may require significant RAM or VRAM.

Tested during development on an NVIDIA DGX Spark workstation.

## Installation

### 1. Install Ollama

Install Ollama on the same machine as napari, then start the local server:

```bash
ollama serve
```

Pull at least one model before using the plugin:

```bash
ollama pull nemotron-cascade-2:30b
```

Optional alternatives:

```bash
ollama pull qwen3-coder-next:latest
ollama pull qwen3.5
ollama pull qwen2.5:7b
```

### 2. Install the plugin

For normal use:

```bash
pip install napari-chat-assistant
```

For development:

```bash
git clone https://github.com/wulinteousa2-hash/napari-chat-assistant.git
cd napari-chat-assistant
pip install -e .
```

## Release

This package is published to PyPI so napari Hub can discover it.

For maintainer release instructions and PyPI publishing setup, see [RELEASING.md](RELEASING.md)
## Usage

1. Start napari.
2. Open `Plugins -> Chat Assistant`.
3. Leave `Base URL` as `http://127.0.0.1:11434` unless your Ollama server is elsewhere.
4. Choose a model from the `Model` dropdown or type a model tag manually.
5. Click `Test`.
6. Start chatting, or use the Library for repeatable tasks and reusable code.

If you already have Python code you want to try, paste it into the Prompt box and click `Run My Code`. This runs viewer-bound code directly inside napari without opening QtConsole.

The assistant works best when prompts describe a concrete action. Natural language is fine.

Examples:
- `Inspect the selected layer`
- `Preview threshold on em_2d_snr_mid`
- `Apply gaussian denoise to em_2d_snr_low with sigma 1.2`
- `Fill holes in mask_messy_2d`
- `Remove small objects from mask_messy_2d with min_size 64`
- `Keep only the largest connected component in mask_messy_2d`
- `Measure labels table for rgb_cells_2d_labels`
- `Create a max intensity projection from em_3d_snr_mid along axis 0`
- `Crop em_2d_snr_high to the bounding box of em_2d_mask with padding 8`
- `Inspect the current ROI`
- `Extract ROI values from em_2d_snr_mid using em_2d_mask`

### Demo Packs

Use the Library `Code` tab to load built-in demo packs for repeatable testing.

Current demo packs include:
- EM 2D SNR sweep
- EM 3D SNR sweep
- RGB cells 2D SNR sweep
- RGB cells 3D SNR sweep
- messy masks 2D/3D

These create named layers so you can test built-in tools quickly without hunting for sample data. Labels layers from the demo packs can also be used as ROIs for ROI inspection and value extraction.

Example pipeline:
1. Run the `EM 2D SNR Sweep` demo pack.
2. `Apply gaussian denoise to em_2d_snr_low with sigma 1.0`
3. `Preview threshold on em_2d_snr_low_gaussian`
4. `Apply threshold now on em_2d_snr_low_gaussian`
5. `Fill holes in em_2d_snr_low_gaussian_labels`
6. `Remove small objects from em_2d_snr_low_gaussian_labels_filled with min_size 64`
7. `Keep only the largest connected component in em_2d_snr_low_gaussian_labels_filled_clean`
8. `Measure mask on em_2d_snr_low_gaussian_labels_filled_clean_largest`

## UI Overview

### Model Connection

- local Ollama base URL
- model picker with discovered local models
- `Test`
- `Use`
- `Setup` help with install, `ollama serve`, and model pull examples
- `Unload`

### Library

- built-in prompts
- built-in demo packs and starter code in the `Code` tab
- recent prompts
- saved prompts
- recent and saved code snippets in the `Code` tab
- pinned prompts
- `saved` keeps your own reusable copy
- `pinned` keeps a prompt at the top regardless of whether it is built-in, recent, or saved
- single click to load
- double click to send from `Prompts` or run from `Code`
- right click to rename or edit tags for saved/recent prompt and code items
- Shift/Ctrl multi-select for batch actions
- `Delete` works on built-in, recent, saved, and code items
- `Clear` keeps saved and pinned items and clears unpinned recent items
- `A-` and `A+` adjust library font size in small steps

### Chat

- multi-line prompt box
- Enter to send
- Shift/Ctrl/Alt+Enter for newline
- transcript showing user messages, assistant replies, tool results, and generated code

### Code Actions

- `Reject`
- `Run Code`
- `Run My Code`
- `Copy Code`
- `Advanced`
- `Help`

`Run Code` is for assistant-generated code that has been staged in the chat.

`Run My Code` is for your own pasted Python from the Prompt box when you want to test or iterate directly inside napari without opening QtConsole.

`Advanced` contains optional integrations such as experimental SAM2 setup and live preview.

### Current Context

- current layer summary from the active napari viewer
- shortened layer names and a compact per-layer summary to avoid over-stretching the left column

### Session

- `Activity` tab shows local status updates, model connection messages, tool execution messages, and code execution/copy actions
- `Telemetry` tab contains the optional telemetry controls
- `Diagnostics` tab provides access to the app log and crash log
- color-highlighted path entries for assistant log, crash log, telemetry log, prompt library, and session memory
- `Enable Telemetry` switch for advanced users
- `Summary`, `Log`, and `Reset` only when telemetry is enabled

## How It Works

The assistant is designed to operate within constrained napari workflows rather than as a general-purpose chatbot.

The current strategy is:
1. collect structured napari viewer context
2. build deterministic per-layer profile objects from the current viewer state
3. add bounded approved session memory when available
4. send that context and the user request to a local Ollama model
5. the model returns a structured JSON response that specifies either:
   - a normal reply
   - a built-in tool call
   - generated Python code
6. run the selected registry-backed tool or expose the generated code through the UI
7. update session memory from explicit user feedback or successful follow-up behavior

This keeps the assistant more grounded than a plain chat interface and makes common operations more reliable.

## Design Direction

The intended architecture is:

1. natural language at the user surface
2. registry-backed tools underneath
3. explicit scope resolution for full-layer and ROI/subregion workflows

This means users should be able to ask for operations in normal language, while the plugin resolves those requests into deterministic tool calls with structured parameters.

### Tool model

Registered tools are the common execution model for:
- chat-triggered actions
- reusable UI actions
- future workflow and pipeline steps
- future plugin-contributed extensions

Each tool is moving toward a shared definition with:
- stable name
- parameter schema
- supported layer types
- prepare/execute/apply lifecycle
- UI metadata
- provenance metadata

### Scope model

For imaging analysis, operations may target:
- the full layer
- a labels mask
- a specific labels object
- a shapes ROI
- a bounding-box crop

Natural language can express these requests, but the plugin still needs deterministic binding rules underneath.

The preferred resolution order is:
1. explicit user binding such as `image_a using roi_shapes`
2. current viewer selection when there is only one clear match
3. a short clarification question when multiple bindings are plausible

Session memory should remain secondary context. Current viewer state and explicit user clarification should remain the primary source of truth.

## Recommended Models

Good starting choices:
- `nemotron-cascade-2:30b`
- `qwen3-coder-next:latest`
- `qwen3.5`
- `qwen2.5:7b`

Selection guidance:
- `nemotron-cascade-2:30b` is the current default and a strong general model for this workflow.
- `qwen3-coder-next:latest` is a better candidate for Python and napari code generation, but it is significantly heavier.
- `qwen3.5` remains a useful alternative general model.
- `qwen2.5:7b` is lighter and may fit smaller-memory systems more easily.

Memory note:
- Larger tags require more RAM or VRAM.
- On the DGX Spark setup used during development, `qwen3-coder-next:latest` may need around 100 GB of available memory to run comfortably.

## Current Limitations

- the dataset profiler is still Phase 1 and remains strongest on already-loaded napari layers rather than reader- or file-format-specific workflows
- TIFF vs OME-Zarr adapter behavior is not implemented yet
- ND2 and Zeiss reader-aware adapters are not implemented in this plugin
- the tool registry is in progress; some tools are now registry-backed, but the migration is not complete yet
- session memory is selective and bounded; it is not full conversation memory
- model output can still be inconsistent, especially when falling back to generated code
- some requests still miss built-in tools and fall through to code generation when a stronger built-in workflow would be preferable
- generated code can still fail if the model invents incorrect napari APIs or unsupported imports
- multi-step workflow planning and replay are not implemented yet
- no image attachment or multimodal input pipeline yet
- performance optimization for very large 2D/3D datasets is still in progress
- hard native crashes in Qt/C-extension code may not be captured cleanly by the plugin crash log even when normal plugin errors are logged

Most reliable current workflow:
- use built-in tools for common layer inspection and mask/image actions
- trust current viewer context and current layer profiles over any remembered prior turn
- use the Library for repeated prompts, demo packs, and reusable code
- use generated code when you want explicit review and control
- use `Run My Code` when you already have working Python and want to test it directly inside napari

For demo and education workflows:
- ask for code that uses the current napari `viewer`
- avoid prompts that create a second `napari.Viewer()` or call `napari.run()`
- prefer docked widgets over unmanaged popup windows for histogram or SNR teaching tools

## Troubleshooting

### Ollama not running

If `Test` fails after restarting your computer, Ollama is usually not running yet.

Start it in a terminal:

```bash
ollama serve
```

Then return to the plugin and click `Test` again.

### Pulling a model

Model downloads are intentionally handled outside the plugin.

To try a different model:
- browse tags at `https://ollama.com/search`
- type the tag into the plugin `Model` field if needed
- pull it in a terminal, for example:

```bash
ollama pull nemotron-cascade-2:30b
```

Then use `Test` to refresh the plugin state.

### Logs and crash logs

The plugin writes two local log files:
- `~/.napari-chat-assistant/assistant.log`
- `~/.napari-chat-assistant/crash.log`

Use these together with the terminal traceback when diagnosing crashes or unclear UI failures.

### Local model telemetry

The plugin also writes lightweight local telemetry to:
- `~/.napari-chat-assistant/model_telemetry.jsonl`

This records real usage events such as:
- request start and completion
- selected model and prompt hash
- total latency
- response type (`reply`, `tool`, `code`, or `error`)
- reject feedback from `👎 Reject`
- approved code execution success or failure

Telemetry is now opt-in from the `Session -> Telemetry` tab through `Enable Telemetry`.

The goal is passive model tracking during actual work rather than separate benchmarking runs.

For advanced users, the `Session -> Telemetry` tab includes:
- `Summary` to generate a quick in-app summary of recent model speed and behavior
- `Log` to inspect the latest raw JSONL records together with the summary
- `Reset` to clear the local telemetry file and start fresh from the next request

This keeps the append-only log intact while making it easier to review without leaving napari.

Generated code is also preflight-validated before execution for common dtype mistakes, unsupported napari imports, and unavailable `viewer.*` APIs. When validation blocks execution, the code remains visible and copyable for review or regeneration.

## Development

Editable install:

```bash
pip install -e .
```

Build a release artifact:

```bash
python -m build
```

## License

MIT. 
