Metadata-Version: 2.5
Name: fossick
Version: 0.1.21
Summary: Web search, fetch, crawl, and browser automation for humans and agents
Project-URL: Repository, https://github.com/vedicreader/fossick
Project-URL: Documentation, https://vedicreader.github.io/fossick/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.12
Requires-Dist: certifi>=2026.7.22
Requires-Dist: curl-cffi>=0.15.0
Requires-Dist: ddgs>=9.14.4
Requires-Dist: diskcache>=5.6.3
Requires-Dist: fastcdp>=0.0.9
Requires-Dist: fastcore>=1.12.31
Requires-Dist: html2text>=2025.4.15
Requires-Dist: mcp<2,>=1.2.0
Requires-Dist: pdf-oxide>=0.3.67
Requires-Dist: pdflite>=0.0.1
Requires-Dist: readability-lxml>=0.8.4.1
Requires-Dist: scrapling[fetchers]>=0.4.8
Requires-Dist: yt-dlp>=2026.3.17
Provides-Extra: dev
Requires-Dist: flashrank>=0.2.10; extra == 'dev'
Requires-Dist: ipykernel>=7.3.0; extra == 'dev'
Requires-Dist: nbdev>=3.3.25; extra == 'dev'
Requires-Dist: notebook>=7.5.5; extra == 'dev'
Description-Content-Type: text/markdown

# fossick


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

fossick gets text off the web and into a program. You give it a search query, a URL, an arXiv id, a YouTube link, a GitHub repository or a local file. It gives back markdown that a person or a language model can read. It was written for AI agents that look things up. Its functions return plain data (strings, lists and dictionaries) that you can print or pass to a model as they are.

The library works in layers. [<code>search</code>](https://vedicreader.github.io/fossick/search.html#search) sends a query to several search engines at once and merges their answers into one ranked list. [`read`](https://vedicreader.github.io/fossick/core.html#read) takes a single target and returns its text. Under [`read`](https://vedicreader.github.io/fossick/core.html#read), [<code>fetch</code>](https://vedicreader.github.io/fossick/core.html#fetch) downloads a page and [`to_md`](https://vedicreader.github.io/fossick/core.html#to_md) turns its HTML into markdown. Many sites refuse plain downloads from scripts. [<code>fetch</code>](https://vedicreader.github.io/fossick/core.html#fetch) can also load a page in a headless browser, in a stealth browser built to pass bot checks, or in a copy of Chrome you have logged in to. For pages that need clicking and typing, [`cdp_connect`](https://vedicreader.github.io/fossick/cdp.html#cdp_connect) drives that Chrome directly. Everything here is also available from the shell as the `fossick` command and as a Model Context Protocol (MCP) server that agents can call.

## Install

``` sh
uv add fossick
```

Searching and plain downloads need nothing else. Rendering JavaScript, the stealth fetcher and [`google`](https://vedicreader.github.io/fossick/search.html#google) run a headless browser through [scrapling](https://github.com/D4Vinci/Scrapling). If those calls raise [`BrowserUnavailable`](https://vedicreader.github.io/fossick/core.html#browserunavailable), run `scrapling install` to download its browsers. The browser automation in `fossick.cdp` drives your installed Google Chrome.

## Quick start

Three calls cover most uses. [<code>search</code>](https://vedicreader.github.io/fossick/search.html#search) returns a list of results, each a dictionary with the page’s `title`, its URL in `href` and a snippet in `body`. `score` is the merged ranking score, and `engines` names the search engines that returned that page. `method='flashrank'` reorders the results with a small relevance model, which needs `pip install flashrank`; without a `method` the merged order is kept.

``` python
results = search('koshas github fts semantic code graph', method='flashrank', n=5)
for r in results: print(r['score'], r['engines'], r['title'], r['href'])
```

    0.030536 ['brave', 'yahoo'] github.com › codegraph-ai › CodeGraphGitHub - codegraph-ai/CodeGraph: CodeGraph builds a semantic... https://github.com/codegraph-ai/CodeGraph
    0.016129 ['yahoo'] vedicreader.github.io › kosha › graphgraph – koshas - vedicreader.github.io https://vedicreader.github.io/kosha/graph.html
    0.015873 ['brave'] fts · GitHub Topics · GitHub https://github.com/topics/fts?o=desc&s=updated
    0.016393 ['yahoo'] pypi.org › project › koshaskoshas · PyPI https://pypi.org/project/koshas/
    0.015625 ['brave'] GitHub - colbymchenry/codegraph: Pre-indexed code knowledge graph, auto syncs on code changes, for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, and Hermes Agent — fewer tokens, fewer tool calls, 100% local https://github.com/colbymchenry/codegraph

``` python
res = research('sqlite WAL mode vs journal mode', n=5)
print(res['digest'][:500])
print([s['href'] for s in res['sources']])
```

    ## Write-Ahead Logging
    https://www.sqlite.org/wal.html

    Write-Ahead Logging

    Table Of Contents

    # 1\. Overview

    The default method by which SQLite implements atomic commit and rollback is a rollback journal. Beginning with version 3.7.0 (2010-07-21), a new "Write-Ahead Log" option (hereafter referred to as "WAL") is available.

    There are advantages and disadvantages to using WAL instead of a rollback journal. Advantages include:

    1. WAL is significantly faster in most scenarios. 
      2. WAL provid
    ['https://www.sqlite.org/wal.html', 'https://blog.sqlite.ai/journal-modes-in-sqlite', 'https://mohit-bhalla.medium.com/understanding-wal-mode-in-sqlite-boosting-performance-in-sql-crud-operations-for-ios-5a8bd8be93d2', 'https://til.simonwillison.net/sqlite/enabling-wal-mode', 'https://fly.io/blog/wal-mode-in-litefs/']

``` python
page = read('https://en.wikipedia.org/wiki/Web_scraping')
print(page.text[:400])
```

    The legality of web scraping varies across the world. In general, web scraping may be against the terms of service of some websites, but the enforceability of these terms is unclear.[11]

    In the United States, website owners can use three major legal claims to prevent undesired web scraping: (1) copyright infringement (compilation), (2) violation of the Computer Fraud and Abuse Act ("CFAA"), and (

## One function for every source

Use [`read`](https://vedicreader.github.io/fossick/core.html#read) when you have a target and want its text. [`read`](https://vedicreader.github.io/fossick/core.html#read) first calls [`what_is`](https://vedicreader.github.io/fossick/core.html#what_is) to classify the target. The result names the reader: `dir` for a local folder, `file` for a local file, `arxiv` for an arXiv URL or id, `youtube` for a YouTube link, `github` for a repository, `ghfile` for a file on GitHub, `pdf` for a URL ending in `.pdf` or `/pdf`, and `web` for any other http or https URL. A path that exists on disk wins over every URL rule.

[`read`](https://vedicreader.github.io/fossick/core.html#read) returns the same seven fields for every supported target:

| field | holds |
|----|----|
| `ok` | True when the reader produced content |
| `kind` | the reader used, one of the names above |
| `title` | the title, or the first heading of the text |
| `source` | the URL, or the absolute path of a local file |
| `text` | the content as markdown |
| `skipped` | why nothing was read, when `ok` is False |
| `meta` | details from that reader, such as the authors of a paper or the HTTP status of a page, and the time it was read |

Here [`read`](https://vedicreader.github.io/fossick/core.html#read) opens this repository’s own README:

``` python
target = str(repo_root()/'README.md')
assert what_is(target) == 'file'
result = read(target)
assert sorted(result) == ['kind', 'meta', 'ok', 'skipped', 'source', 'text', 'title']
assert (result.ok, result.kind, result.title) == (True, 'file', 'fossick')
assert '# fossick' in result.text and result.skipped is None
result.kind, result.title, result.ok
```

A folder or a GitHub repository is a tree of files, not one document. [`read`](https://vedicreader.github.io/fossick/core.html#read) does not turn it into text. It returns `ok=True` with an empty `text` and the local path in `meta['path']`. The caller chooses which files to open. For a GitHub repository, that path is a fresh clone in fossick’s cache. With `pages=True`, a PDF returns a list of `(page_index, text)` pairs. Page indexes start at 0.

Sometimes a target is recognised but yields nothing: a site serves a bot check instead of the page, a YouTube video has no English captions, or a URL that looked like a PDF returns something else. [`read`](https://vedicreader.github.io/fossick/core.html#read) reports these with `ok=False` and a reason in `skipped`, and does not raise. It raises `ValueError` only when the target is none of the kinds above.

## Modules

| notebook | what it covers |
|----|----|
| `00_core` | downloading pages ([<code>fetch</code>](https://vedicreader.github.io/fossick/core.html#fetch), [<code>crawl</code>](https://vedicreader.github.io/fossick/core.html#crawl)), HTML to markdown ([`to_md`](https://vedicreader.github.io/fossick/core.html#to_md)), the readers for arXiv, GitHub, YouTube and PDFs, finding the JSON API behind a page, and [`read`](https://vedicreader.github.io/fossick/core.html#read) |
| `01_cdp` | driving Chrome through the DevTools protocol: page snapshots for agents, filling forms, running a list of steps, capturing network requests |
| `02_search` | web search across several engines, merging and reranking results, [`google`](https://vedicreader.github.io/fossick/search.html#google), and [<code>research</code>](https://vedicreader.github.io/fossick/search.html#research) |
| `03_cli` | the `fossick` shell command |
| `04_mcp` | the MCP server that offers fossick’s functions to coding agents |
| `05_shop` | adding products to a shopping cart and filling checkout forms on a live store |
| `06_quality` | choosing which search results are worth reading: removing copycat sites, preferring authoritative ones, and splitting a long question into several searches |

## Common calls

``` python
search('query', n=10, region='auto', timelimit='y')   # results from the past year
research('query', n=5)                    # search, read the top pages, return a digest that cites them
fetch(url, auto=True)                     # try stronger download methods until one is not blocked
crawl(url, max_pages=5, same_domain=True) # follow links from url, staying on its site
google('query', n=10)                     # Google's own ranking, through the stealth browser
cdp = await cdp_connect(); pg = await cdp.open_page(url); print(await pg.snapshot())
```

From the shell: `fossick search "q"`, `fossick fetch URL --auto`, `fossick research "q"`. For agents: `uvx fossick-mcp`, or `fossick-mcp --http` to serve over HTTP instead of standard input and output.

## A logged-in Chrome

Some pages are only shown to a logged-in user. fossick reaches them through a separate copy of Chrome that runs with remote debugging switched on, on port 9223, with its own profile in `~/.cache/fastcdp/cdp-chrome`. [`cdp_connect`](https://vedicreader.github.io/fossick/cdp.html#cdp_connect) starts it the first time it is needed. Log in to a site once in that window. The cookies stay in the profile. Later calls such as `fetch(url, session=True)` then use the logged-in session. `fetch(url, auto=True)` and [`read`](https://vedicreader.github.io/fossick/core.html#read) fall back to this Chrome when every other method is blocked.

To start it by hand in a visible window, for logging in, run `await cdp_setup(headless=False)`. The `fastcdp-setup` command, installed with fossick’s `fastcdp` dependency, adds a launcher for the same Chrome and profile to your applications.
