Metadata-Version: 2.5
Name: sift-downloads
Version: 0.1.1
Summary: Ask your Downloads folder where a file is — and what it says. Runs entirely on your machine.
Project-URL: Homepage, https://github.com/Dheemant-Dixit/sift
Project-URL: Repository, https://github.com/Dheemant-Dixit/sift
Project-URL: Issues, https://github.com/Dheemant-Dixit/sift/issues
Author-email: Dheemant Dixit <dheemantdixit1107@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Dheemant Dixit
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: downloads,local-first,ollama,rag,search,semantic-search
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Desktop Environment :: File Managers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Requires-Dist: litellm>=1.60
Requires-Dist: numpy>=1.24
Requires-Dist: prompt-toolkit>=3.0
Requires-Dist: pypdf>=4.0
Requires-Dist: python-docx>=1.1
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: watch
Requires-Dist: watchdog>=4.0; extra == 'watch'
Description-Content-Type: text/markdown

# sift

**You downloaded it. You can't find it. And when you do, it's forty pages.**

sift searches your Downloads folder by meaning, not just by filename, and
answers questions about what's in there. Everything runs on your own machine —
no API keys, no accounts, no documents uploaded.

*Installs as `sift-downloads` (plain `sift` was taken on PyPI). The command you
type is `sift`.*

```
● sift  /Users/you/Downloads

> rental agreement

   1. ScannedRentalAgreement.pdf                    0.98 name
      2.4MB · 3mo ago
      (no extractable text (scanned or image-only?))
   2. lease-notes.md                                0.64 ·
      217B · 7mo ago
      # Lease terms The notice period for terminating this lease is 60…

> ?what is the notice period and the deposit

  Notice period: 60 days [lease-notes.md]
  Security deposit: 100,000 INR, refundable within 30 days [lease-notes.md]

┌──────────────────────────| sift |──────────────────────────┐
│>                                                           │
└────────────────────────────────────────────────────────────┘
```

Type anything to search. Start with `?` to ask a question instead. The prompt
stays at the bottom and results scroll up into your normal terminal history, so
nothing takes over your screen and you keep your scrollback.

---

## Install

```bash
# 1. a local model runner (free, but ~5GB of models to download)
brew install ollama && brew services start ollama     # macOS
# curl -fsSL https://ollama.com/install.sh | sh       # Linux

ollama pull nomic-embed-text     # 274MB — turns text into vectors
ollama pull llama3.1:8b          # 4.9GB — writes the answers

# 2. sift  (the command is `sift`; the package name has a suffix because
#           plain `sift` was already taken on PyPI)
pip install "sift-downloads[watch]"

# 3. read your Downloads folder (a few hundred files takes under a minute)
sift index

# 4. go
sift
```

If anything looks wrong, run `sift doctor`. It checks each piece and prints the
exact command to fix whatever is broken.

**On a small disk or a slow connection?** The 4.9GB one is only used to write
the final answer. Swap it for something smaller — `sift` still finds files just
as well, and answers get a little blunter:

```bash
ollama pull llama3.2:3b
sift ask "..." --chat-model ollama_chat/llama3.2:3b    # or SIFT_CHAT_MODEL
```

The 274MB embedding model is the one that does the searching, and it is not
optional.

---

## Using it

Run `sift` on its own for the interactive session above:

| Type this | What happens |
|---|---|
| `rental agreement` | searches for it |
| `?what is my notice period` | asks a question, answered from your files |
| `/open 2` `/reveal 2` | opens result 2, or shows it in your file manager |
| `/find -r invoice` | searches, preferring recently downloaded files |
| `/sync` `/status` `/help` | update the index, see what's indexed, list commands |
| `ctrl-d` | quit |

Or run single commands without the session:

| Command | What it does |
|---|---|
| `sift find "bank statement"` | ranked list of files; add `--open 1` to open one |
| `sift ask "what's my policy number?"` | one grounded answer with sources |
| `sift index` | update the index (usually under a second) |
| `sift index --rebuild` | start over; needed after changing models |
| `sift unlock` | read your password-protected PDFs (asks for each password) |
| `sift status` | what's indexed, and what was skipped and why |
| `sift search "query"` | raw passage scores, for tuning |
| `sift watch` | keep the index updated as the folder changes |
| `sift purge` | delete the index (your files are untouched) |

`find` and `ask` update the index before running, so results are never stale.
Use `--no-sync` to skip that.

### Why it finds files it can't read

A scanned PDF is a picture. There is no text inside to search. Same for a `.zip`
or a `.dmg`.

Those are exactly the files people lose, so sift keeps a record of **every** file
it sees and matches on filenames as well as contents. That is why
`ScannedRentalAgreement.pdf` is the top result above: sift cannot read a word of
it, and still finds it.

Each result shows how it matched — `·` means the contents matched, `name` means
the filename did. Identical copies like `Statement (1).pdf` collapse into one
result that tells you about its twins.

### Password-protected PDFs

Banks send statements locked with a password. sift tells you which files those
are, instead of guessing that they're scanned:

```
$ sift status
2 file(s) — password-protected (findable by name only):
  · AccountStatement_40871876782.pdf
  · lony3005_00000040871876782_E.pdf
  → sift unlock     to read these
```

`sift unlock` asks for each password, reads the file, and adds it to the index.
**The password is never stored** — not in a file, not in your keychain — so
`sift index --rebuild` will ask again. That is deliberate. Two things worth
knowing before you run it:

- Unlocking puts that document's text into the index, which is **not** encrypted.
  A file you locked on purpose becomes readable in `index.npz`.
- Some PDFs are locked only to stop printing or copying, and open with an empty
  password. sift tries that first, so those never reach you as a prompt.

---

## Privacy

Your Downloads folder holds bank statements, ID scans and contracts. So:

- **No document text leaves your machine.** Both models run locally through
  Ollama, so every byte of every file you index is read, embedded and answered
  on `localhost`.
- **Not even a phone-home.** sift talks to models through
  [litellm](https://github.com/BerriAI/litellm), which by default downloads a
  public price list of known models from `raw.githubusercontent.com` when it
  loads. That request carries nothing about you, but it is still a request, so
  sift turns it off (`LITELLM_LOCAL_MODEL_COST_MAP`) and uses the copy shipped
  inside the package. sift does no cost accounting and never reads that list.

  The result is that a default run opens **no connection except to Ollama on
  `localhost`**. Don't take our word for it — `lsof -i`, Little Snitch or
  `tcpdump` will tell you. Please check.
- **The index holds the actual text of your documents.** It lives in your
  system's user-data folder — `sift status` prints the path. Don't commit it or
  share the `.npz`. `sift purge` deletes it. This includes anything you
  `sift unlock` — that text is stored in the clear like everything else.
- **Cloud models need explicit permission.** sift can use Anthropic, OpenAI or
  Gemini through [litellm](https://github.com/BerriAI/litellm), but naming a
  cloud model is not enough. It refuses without `--allow-cloud` and then warns
  you each session. Sending your documents to someone else's server should be a
  decision, not a side effect of editing a config value.

```bash
export ANTHROPIC_API_KEY=sk-...
sift ask "..." --chat-model anthropic/claude-sonnet-4-5 --allow-cloud
```

The two models are separate settings. You can keep embeddings local, so your
whole folder stays home, and use a cloud model only to write the final answer
from the few passages retrieved.

---

## Settings

Nothing needs configuring. Everything can be. A CLI flag beats an environment
variable, which beats `.env`, which beats the default.

| Setting | Flag | Env var | Default |
|---|---|---|---|
| Folder to search | `--source` | `SIFT_SOURCE` | your Downloads folder |
| Where the index lives | `--data-dir` | `SIFT_DATA_DIR` | system user-data folder |
| Embedding model | `--embed-model` | `SIFT_EMBED_MODEL` | `ollama/nomic-embed-text` |
| Answering model | `--chat-model` | `SIFT_CHAT_MODEL` | `ollama_chat/llama3.1:8b` |
| Chunk size / overlap | `--chunk-size` / `--chunk-overlap` | `SIFT_CHUNK_SIZE` / `SIFT_CHUNK_OVERLAP` | 1000 / 150 |
| Passages per answer | `--top-k` | `SIFT_TOP_K` | 5 |
| Relevance bar for `ask` | `--min-score` | `SIFT_MIN_SCORE` | 0.55 |
| Candidate bar for `find` | — | `SIFT_FIND_MIN_SCORE` | 0.40 |
| Largest file to read | `--max-file-mb` | `SIFT_MAX_FILE_MB` | 50 |
| Allow cloud models | `--allow-cloud` | `SIFT_ALLOW_CLOUD` | off |

sift is not limited to Downloads — `sift find "..." --source ~/Documents` works
fine. Downloads is just where this problem actually bites.

See [`.env.example`](.env.example).

**A warning about `--min-score`.** The default 0.55 was measured for
`nomic-embed-text` on one particular set of documents. It does not transfer. If
you change embedding models, work out your own — see
[the calibration guide](docs/DESIGN.md#calibrating-the-relevance-bar).

---

## Keeping the index fresh

The index updates incrementally. sift records each file's size and modification
time, so a sync only re-reads what actually changed — about a second, against
forty for a full rebuild. That is why `find` and `ask` can afford to sync before
every query.

If you want it updated in the background, [`contrib/`](contrib/) has ready-made
launchd and systemd files. They are documented, not installed for you. sift
never writes to your system.

---

## What it can't do

- **No OCR.** Scanned PDFs give up no text, so their contents can't be searched.
  They stay findable by filename. (A *locked* PDF is a different problem with a
  real fix — see `sift unlock` above.)
- **No re-ranker.** Ranking is by topic similarity, not by "does this answer the
  question". Ask "what is my designation?" and a dozen employment documents can
  outrank the payslip that says it outright.
- **Top level only.** sift doesn't walk into subfolders, on purpose — one
  unzipped project would drag in thousands of files.
- **Answers aren't guaranteed correct.** A small local model can still drift past
  its instructions. The one hard rule is that if nothing relevant is found, sift
  refuses without calling the model at all.

The [design notes](docs/DESIGN.md#limitations) go into why, and what would fix
each one.

---

## How it works

A RAG pipeline built from scratch — no LangChain, no vector database. Text is
split into overlapping chunks, each chunk becomes a vector, and search is one dot
product against a matrix of unit vectors.

**→ [Design notes](docs/DESIGN.md)** — the pipeline, which file to read first,
why the vector store is shaped the way it is, how to calibrate the relevance
bar, and the full limitations.

## Development

```bash
pip install -e ".[watch,dev]"
pytest
```

The tests need no Ollama and touch no real folder.

## License

MIT — see [LICENSE](LICENSE).
