Metadata-Version: 2.1
Name: mashpit
Version: 1.0.0
Summary: A sketch-based surveillance platform
Author-email: Tongzhou Xu <tongzhou.xu@uga.edu>
License:                     GNU GENERAL PUBLIC LICENSE
                               Version 2, June 1991
        
         Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
         51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
         Everyone is permitted to copy and distribute verbatim copies
         of this license document, but changing it is not allowed.
        
                                    Preamble
        
          The licenses for most software are designed to take away your
        freedom to share and change it.  By contrast, the GNU General Public
        License is intended to guarantee your freedom to share and change free
        software--to make sure the software is free for all its users.  This
        General Public License applies to most of the Free Software
        Foundation's software and to any other program whose authors commit to
        using it.  (Some other Free Software Foundation software is covered by
        the GNU Lesser General Public License instead.)  You can apply it to
        your programs, too.
        
          When we speak of free software, we are referring to freedom, not
        price.  Our General Public Licenses are designed to make sure that you
        have the freedom to distribute copies of free software (and charge for
        this service if you wish), that you receive source code or can get it
        if you want it, that you can change the software or use pieces of it
        in new free programs; and that you know you can do these things.
        
          To protect your rights, we need to make restrictions that forbid
        anyone to deny you these rights or to ask you to surrender the rights.
        These restrictions translate to certain responsibilities for you if you
        distribute copies of the software, or if you modify it.
        
          For example, if you distribute copies of such a program, whether
        gratis or for a fee, you must give the recipients all the rights that
        you have.  You must make sure that they, too, receive or can get the
        source code.  And you must show them these terms so they know their
        rights.
        
          We protect your rights with two steps: (1) copyright the software, and
        (2) offer you this license which gives you legal permission to copy,
        distribute and/or modify the software.
        
          Also, for each author's protection and ours, we want to make certain
        that everyone understands that there is no warranty for this free
        software.  If the software is modified by someone else and passed on, we
        want its recipients to know that what they have is not the original, so
        that any problems introduced by others will not reflect on the original
        authors' reputations.
        
          Finally, any free program is threatened constantly by software
        patents.  We wish to avoid the danger that redistributors of a free
        program will individually obtain patent licenses, in effect making the
        program proprietary.  To prevent this, we have made it clear that any
        patent must be licensed for everyone's free use or not licensed at all.
        
          The precise terms and conditions for copying, distribution and
        modification follow.
        
                            GNU GENERAL PUBLIC LICENSE
           TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
        
          0. This License applies to any program or other work which contains
        a notice placed by the copyright holder saying it may be distributed
        under the terms of this General Public License.  The "Program", below,
        refers to any such program or work, and a "work based on the Program"
        means either the Program or any derivative work under copyright law:
        that is to say, a work containing the Program or a portion of it,
        either verbatim or with modifications and/or translated into another
        language.  (Hereinafter, translation is included without limitation in
        the term "modification".)  Each licensee is addressed as "you".
        
        Activities other than copying, distribution and modification are not
        covered by this License; they are outside its scope.  The act of
        running the Program is not restricted, and the output from the Program
        is covered only if its contents constitute a work based on the
        Program (independent of having been made by running the Program).
        Whether that is true depends on what the Program does.
        
          1. You may copy and distribute verbatim copies of the Program's
        source code as you receive it, in any medium, provided that you
        conspicuously and appropriately publish on each copy an appropriate
        copyright notice and disclaimer of warranty; keep intact all the
        notices that refer to this License and to the absence of any warranty;
        and give any other recipients of the Program a copy of this License
        along with the Program.
        
        You may charge a fee for the physical act of transferring a copy, and
        you may at your option offer warranty protection in exchange for a fee.
        
          2. You may modify your copy or copies of the Program or any portion
        of it, thus forming a work based on the Program, and copy and
        distribute such modifications or work under the terms of Section 1
        above, provided that you also meet all of these conditions:
        
            a) You must cause the modified files to carry prominent notices
            stating that you changed the files and the date of any change.
        
            b) You must cause any work that you distribute or publish, that in
            whole or in part contains or is derived from the Program or any
            part thereof, to be licensed as a whole at no charge to all third
            parties under the terms of this License.
        
            c) If the modified program normally reads commands interactively
            when run, you must cause it, when started running for such
            interactive use in the most ordinary way, to print or display an
            announcement including an appropriate copyright notice and a
            notice that there is no warranty (or else, saying that you provide
            a warranty) and that users may redistribute the program under
            these conditions, and telling the user how to view a copy of this
            License.  (Exception: if the Program itself is interactive but
            does not normally print such an announcement, your work based on
            the Program is not required to print an announcement.)
        
        These requirements apply to the modified work as a whole.  If
        identifiable sections of that work are not derived from the Program,
        and can be reasonably considered independent and separate works in
        themselves, then this License, and its terms, do not apply to those
        sections when you distribute them as separate works.  But when you
        distribute the same sections as part of a whole which is a work based
        on the Program, the distribution of the whole must be on the terms of
        this License, whose permissions for other licensees extend to the
        entire whole, and thus to each and every part regardless of who wrote it.
        
        Thus, it is not the intent of this section to claim rights or contest
        your rights to work written entirely by you; rather, the intent is to
        exercise the right to control the distribution of derivative or
        collective works based on the Program.
        
        In addition, mere aggregation of another work not based on the Program
        with the Program (or with a work based on the Program) on a volume of
        a storage or distribution medium does not bring the other work under
        the scope of this License.
        
          3. You may copy and distribute the Program (or a work based on it,
        under Section 2) in object code or executable form under the terms of
        Sections 1 and 2 above provided that you also do one of the following:
        
            a) Accompany it with the complete corresponding machine-readable
            source code, which must be distributed under the terms of Sections
            1 and 2 above on a medium customarily used for software interchange; or,
        
            b) Accompany it with a written offer, valid for at least three
            years, to give any third party, for a charge no more than your
            cost of physically performing source distribution, a complete
            machine-readable copy of the corresponding source code, to be
            distributed under the terms of Sections 1 and 2 above on a medium
            customarily used for software interchange; or,
        
            c) Accompany it with the information you received as to the offer
            to distribute corresponding source code.  (This alternative is
            allowed only for noncommercial distribution and only if you
            received the program in object code or executable form with such
            an offer, in accord with Subsection b above.)
        
        The source code for a work means the preferred form of the work for
        making modifications to it.  For an executable work, complete source
        code means all the source code for all modules it contains, plus any
        associated interface definition files, plus the scripts used to
        control compilation and installation of the executable.  However, as a
        special exception, the source code distributed need not include
        anything that is normally distributed (in either source or binary
        form) with the major components (compiler, kernel, and so on) of the
        operating system on which the executable runs, unless that component
        itself accompanies the executable.
        
        If distribution of executable or object code is made by offering
        access to copy from a designated place, then offering equivalent
        access to copy the source code from the same place counts as
        distribution of the source code, even though third parties are not
        compelled to copy the source along with the object code.
        
          4. You may not copy, modify, sublicense, or distribute the Program
        except as expressly provided under this License.  Any attempt
        otherwise to copy, modify, sublicense or distribute the Program is
        void, and will automatically terminate your rights under this License.
        However, parties who have received copies, or rights, from you under
        this License will not have their licenses terminated so long as such
        parties remain in full compliance.
        
          5. You are not required to accept this License, since you have not
        signed it.  However, nothing else grants you permission to modify or
        distribute the Program or its derivative works.  These actions are
        prohibited by law if you do not accept this License.  Therefore, by
        modifying or distributing the Program (or any work based on the
        Program), you indicate your acceptance of this License to do so, and
        all its terms and conditions for copying, distributing or modifying
        the Program or works based on it.
        
          6. Each time you redistribute the Program (or any work based on the
        Program), the recipient automatically receives a license from the
        original licensor to copy, distribute or modify the Program subject to
        these terms and conditions.  You may not impose any further
        restrictions on the recipients' exercise of the rights granted herein.
        You are not responsible for enforcing compliance by third parties to
        this License.
        
          7. If, as a consequence of a court judgment or allegation of patent
        infringement or for any other reason (not limited to patent issues),
        conditions are imposed on you (whether by court order, agreement or
        otherwise) that contradict the conditions of this License, they do not
        excuse you from the conditions of this License.  If you cannot
        distribute so as to satisfy simultaneously your obligations under this
        License and any other pertinent obligations, then as a consequence you
        may not distribute the Program at all.  For example, if a patent
        license would not permit royalty-free redistribution of the Program by
        all those who receive copies directly or indirectly through you, then
        the only way you could satisfy both it and this License would be to
        refrain entirely from distribution of the Program.
        
        If any portion of this section is held invalid or unenforceable under
        any particular circumstance, the balance of the section is intended to
        apply and the section as a whole is intended to apply in other
        circumstances.
        
        It is not the purpose of this section to induce you to infringe any
        patents or other property right claims or to contest validity of any
        such claims; this section has the sole purpose of protecting the
        integrity of the free software distribution system, which is
        implemented by public license practices.  Many people have made
        generous contributions to the wide range of software distributed
        through that system in reliance on consistent application of that
        system; it is up to the author/donor to decide if he or she is willing
        to distribute software through any other system and a licensee cannot
        impose that choice.
        
        This section is intended to make thoroughly clear what is believed to
        be a consequence of the rest of this License.
        
          8. If the distribution and/or use of the Program is restricted in
        certain countries either by patents or by copyrighted interfaces, the
        original copyright holder who places the Program under this License
        may add an explicit geographical distribution limitation excluding
        those countries, so that distribution is permitted only in or among
        countries not thus excluded.  In such case, this License incorporates
        the limitation as if written in the body of this License.
        
          9. The Free Software Foundation may publish revised and/or new versions
        of the General Public License from time to time.  Such new versions will
        be similar in spirit to the present version, but may differ in detail to
        address new problems or concerns.
        
        Each version is given a distinguishing version number.  If the Program
        specifies a version number of this License which applies to it and "any
        later version", you have the option of following the terms and conditions
        either of that version or of any later version published by the Free
        Software Foundation.  If the Program does not specify a version number of
        this License, you may choose any version ever published by the Free Software
        Foundation.
        
          10. If you wish to incorporate parts of the Program into other free
        programs whose distribution conditions are different, write to the author
        to ask for permission.  For software which is copyrighted by the Free
        Software Foundation, write to the Free Software Foundation; we sometimes
        make exceptions for this.  Our decision will be guided by the two goals
        of preserving the free status of all derivatives of our free software and
        of promoting the sharing and reuse of software generally.
        
                                    NO WARRANTY
        
          11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
        FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
        OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
        PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
        OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
        MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
        TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
        PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
        REPAIR OR CORRECTION.
        
          12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
        WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
        REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
        INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
        OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
        TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
        YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
        PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
        POSSIBILITY OF SUCH DAMAGES.
        
                             END OF TERMS AND CONDITIONS
        
                    How to Apply These Terms to Your New Programs
        
          If you develop a new program, and you want it to be of the greatest
        possible use to the public, the best way to achieve this is to make it
        free software which everyone can redistribute and change under these terms.
        
          To do so, attach the following notices to the program.  It is safest
        to attach them to the start of each source file to most effectively
        convey the exclusion of warranty; and each file should have at least
        the "copyright" line and a pointer to where the full notice is found.
        
            <one line to give the program's name and a brief idea of what it does.>
            Copyright (C) <year>  <name of author>
        
            This program is free software; you can redistribute it and/or modify
            it under the terms of the GNU General Public License as published by
            the Free Software Foundation; either version 2 of the License, or
            (at your option) any later version.
        
            This program is distributed in the hope that it will be useful,
            but WITHOUT ANY WARRANTY; without even the implied warranty of
            MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
            GNU General Public License for more details.
        
            You should have received a copy of the GNU General Public License along
            with this program; if not, write to the Free Software Foundation, Inc.,
            51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
        
        Also add information on how to contact you by electronic and paper mail.
        
        If the program is interactive, make it output a short notice like this
        when it starts in an interactive mode:
        
            Gnomovision version 69, Copyright (C) year name of author
            Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
            This is free software, and you are welcome to redistribute it
            under certain conditions; type `show c' for details.
        
        The hypothetical commands `show w' and `show c' should show the appropriate
        parts of the General Public License.  Of course, the commands you use may
        be called something other than `show w' and `show c'; they could even be
        mouse-clicks or menu items--whatever suits your program.
        
        You should also get your employer (if you work as a programmer) or your
        school, if any, to sign a "copyright disclaimer" for the program, if
        necessary.  Here is a sample; alter the names:
        
          Yoyodyne, Inc., hereby disclaims all copyright interest in the program
          `Gnomovision' (which makes passes at compilers) written by James Hacker.
        
          <signature of Ty Coon>, 1 April 1989
          Ty Coon, President of Vice
        
        This General Public License does not permit incorporating your program into
        proprietary programs.  If your program is a subroutine library, you may
        consider it more useful to permit linking proprietary applications with the
        library.  If this is what you want to do, use the GNU Lesser General
        Public License instead of this License.
        
Keywords: sourmash,surveillance,pathogen
Requires-Python: <3.12,>=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sourmash~=4.6.1
Requires-Dist: screed<1.1.3,>=1.0.5
Requires-Dist: pandas~=2.0.3
Requires-Dist: numpy~=1.24.4
Requires-Dist: pyarrow<26,>=7
Requires-Dist: biopython~=1.83
Requires-Dist: scikit-bio~=0.6.2
Requires-Dist: tqdm~=4.67.0
Requires-Dist: matplotlib~=3.7.5
Requires-Dist: streamlit~=1.40.1

# Mashpit

[![DOI](https://joss.theoj.org/papers/10.21105/joss.07306/status.svg)](https://doi.org/10.21105/joss.07306)
[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat)](http://bioconda.github.io/recipes/mashpit/README.html)
![unittest](https://github.com/tongzhouxu/mashpit/actions/workflows/python-app.yml/badge.svg)
[![License: GPL v2](https://img.shields.io/badge/License-GPL_v2-blue.svg)](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html)
[![PyPI release](https://img.shields.io/pypi/v/mashpit)](https://pypi.python.org/pypi/mashpit/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

**Mashpit** is a lightweight, local genomic epidemiology platform for rapidly comparing a query genome against representative genomes from NCBI Pathogen Detection SNP clusters.

Mashpit is designed for offline or resource-limited analysis, including deployment on small computers such as a Raspberry Pi. It combines compact genomic sketches, epidemiological metadata, and tree-guided representative selection to support fast preliminary genome screening without requiring a centralized computing service.

**Mashpit 1.0** provides adaptive tree-based representative selection, local database management, and reproducible screening outputs. See the [changelog](CHANGELOG.md) for changes since 0.9.10. Use Python 3.8–3.11; Python 3.11 is recommended.

### Mashpit and IsoScout

Mashpit builds and manages sketch databases and screens assembled genomes locally.
[IsoScout](https://github.com/tongzhouxu/IsoScout) uses Mashpit screening as the
first stage of a broader workflow with input QC, candidate selection, SNP
comparison, and optional cluster-member expansion. Mashpit's similarity scores
and near-top flags prioritize candidates; they do not establish outbreak
membership or replace SNP analysis.

Existing IsoScout `databases-v2` packages can supply exact-release cluster
membership to Mashpit reports. Taxon merges preserve the optional `REP_CLUSTER`
annotation table and reject conflicting labels. IsoScout containers pin their
own Mashpit revision: installing Mashpit 1.0 on the host does not update a
container. Follow IsoScout's runtime and database-manifest requirements when
integrating a new or merged database.

See the [1.0 validation record](docs/release-1.0-validation.md) for the tested
IsoScout revision, real-genome smoke checks, and their limits. These checks do
not establish biological accuracy or validate the full IsoScout SNP workflow.

## Key features

- **Rapid local queries:** Compare an assembled genome against a compact database of representative pathogen genomes.
- **Tree-based representative selection:** Select representatives directly from NCBI Pathogen Detection SNP trees rather than relying on a single centroid for every cluster.
- **Adaptive database size:** Use more representatives for large or genetically diverse SNP clusters and fewer representatives for compact clusters.
- **Optional radius control:** Limit the maximum tree distance between cluster members and their selected representatives.
- **Robust assembly retrieval:** Exclude records without assembly accessions, retry failed genome downloads, and remove persistently unavailable representatives.
- **Integrated metadata:** Return epidemiological information such as isolation date, location, host, BioSample accession, assembly accession, and SNP-cluster identifiers.
- **Local visualization:** Produce a query result table and a tree showing the relationship between the query and its closest database representatives.
- **Offline operation:** After a database has been built, routine queries can be performed without uploading genomic data.
- **Custom databases:** Build a database directly from local FASTA files, with no NCBI download at all - useful for proprietary or unpublished sequence collections.
- **Extensible metadata:** Add your own custom columns to any existing database - taxon, accession, or custom - with a built-in, per-database change history.
- **Parallel database loading:** Large databases are automatically sharded so `mashpit query` loads and compares them across CPU cores instead of single-threaded.

## Mashpit 1.0 database design

An NCBI Pathogen Detection taxon database contains many SNP clusters. A single representative is not sufficient for every cluster because cluster size and within-cluster diversity vary substantially.

Mashpit 1.0 therefore uses an adaptive tree-based strategy:

1. Download the selected NCBI Pathogen Detection metadata and SNP trees.
2. Remove records that do not have a valid assembly accession.
3. Determine the number of representatives required for each SNP cluster from its tree structure.
4. Select representative leaves that cover the cluster, optionally subject to a user-defined tree radius.
5. Download the selected assemblies.
6. Retry failed downloads for a limited number of attempts.
7. Remove representatives whose assemblies remain unavailable.
8. Build the sketch database and retain metadata linking every representative to its SNP cluster.

This approach reduces redundancy in compact clusters while preserving multiple genomic backgrounds in large or diverse clusters.

## Installation

Mashpit currently declares support for Python `>=3.8,<3.12`; use Python **3.11** for the Raspberry Pi instructions below. Python 3.12 and newer are not currently supported.

### Option 1: Conda or Mamba

Bioconda packaging can lag behind GitHub and PyPI. Use the pinned pip command
below for 1.0.0; the following installs the version currently available on Bioconda:

```bash
conda create -n mashpit -c conda-forge -c bioconda mashpit
conda activate mashpit
```

### Option 2: pip

Mashpit requires the NCBI Datasets command-line tool for downloading assemblies when building taxon or accession databases. It is not required for querying an existing database or building a custom database from local FASTA files.

For Intel/AMD 64-bit Linux (`x86_64`), install it with the following commands. Raspberry Pi users should follow the [ARM64 instructions below](#raspberry-pi-git-clone-and-a-python-virtual-environment).

```bash
mkdir -p "$HOME/.local/bin"
curl -fL -o "$HOME/.local/bin/datasets" \
  'https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/v2/linux-amd64/datasets'
chmod +x "$HOME/.local/bin/datasets"
export PATH="$HOME/.local/bin:$PATH"
```

Install the stable release:

```bash
python -m pip install 'mashpit==1.0.0'
```

Install the development version from source:

```bash
git clone https://github.com/tongzhouxu/mashpit.git
cd mashpit
python -m pip install .
```

Verify the installation:

```bash
mashpit --help
```

### Raspberry Pi: Git clone and a Python virtual environment

Use a **64-bit operating system** with **Python 3.11**. These instructions install the current Git source without Conda and assume a Debian-based system such as Raspberry Pi OS, with `python3` pointing to Python 3.11.

Check the architecture, user-space bitness, and Python version first:

```bash
uname -m
getconf LONG_BIT
python3 --version
```

The expected results are `aarch64`, `64`, and `Python 3.11.x`. If the system uses 32-bit user space, use a 64-bit OS for this installation route. If `python3` is 3.12 or newer, install Python 3.11 with its matching `venv` and development packages separately, then use `python3.11 -m venv .venv` below. Creating a virtual environment does not change the interpreter version; do not replace the OS's system Python.

Install the system packages needed to create the environment and compile dependencies:

```bash
sudo apt update
sudo apt install -y git curl ca-certificates build-essential pkg-config \
  python3-venv python3-dev libhdf5-dev
```

Clone the repository, create and activate the virtual environment, and install Mashpit:

```bash
git clone https://github.com/tongzhouxu/mashpit.git
cd mashpit

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install .

python -m pip check
mashpit --help
```

Some dependencies compile from source, so installation can take time on a Pi. Keep the virtual environment activated while installing and running Mashpit; do not use `sudo pip`.

If you will build taxon or accession databases that download NCBI assemblies, install the **Linux ARM64** Datasets executable into the activated virtual environment:

```bash
curl -fL -o "$VIRTUAL_ENV/bin/datasets" \
  'https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/v2/linux-arm64/datasets'
chmod +x "$VIRTUAL_ENV/bin/datasets"
datasets --version
```

The `linux-amd64` executable in the general Linux instructions is for Intel/AMD CPUs and will not run natively on a Pi. See the [NCBI Datasets installation documentation](https://www.ncbi.nlm.nih.gov/datasets/docs/v2/command-line-tools/download-and-install/) for platform downloads. Skip this step if you only query existing databases or build custom databases from local FASTA files.

In a new terminal, change to your cloned repository and reactivate the environment:

```bash
cd /path/to/mashpit
source .venv/bin/activate
```

To update an existing installation from Git, run these commands from the repository with the environment activated:

```bash
git pull --ff-only
python -m pip install .
python -m pip check
```

The repository does not currently run ARM-specific CI tests; these instructions are not a claim of validation on Raspberry Pi hardware.

## Database contents

A Mashpit database is stored in a directory containing the sketch database and its metadata. Depending on the database type, it includes files such as:

```text
DATABASE_NAME/
├── DATABASE_NAME.db               # sqlite metadata and representative tables
├── DATABASE_NAME.sig              # merged sourmash signatures (a single file for smaller
│                                   #   databases, or a directory of shards for larger ones -
│                                   #   see "Sharding a database for faster loading")
├── representatives.tsv            # taxon databases only
├── cluster_summary.tsv            # taxon databases only
├── membership/                    # complete source-release TSVs for taxon databases
│   ├── PDG_VERSION.metadata.tsv.gz
│   └── PDG_VERSION.reference_target.all_isolates.tsv.gz
├── unavailable_assemblies.tsv     # taxon/accession databases: accessions that could not be downloaded
└── unavailable_sequences.tsv      # custom databases: local FASTA files that failed to sketch
```

The build log itself (`mashpit-<timestamp>.log`) is written to the current working directory alongside the database folder, not inside it.

Keep the entire database directory together when moving it to another computer.

Taxon builds retain the full metadata and cluster-isolate membership from the
same resolved PDG release, including isolates without a downloadable assembly.
The SQLite `CLUSTER_MEMBERS` table contains the fields used by cluster reports;
the compressed TSVs preserve all original columns for later use and candidate
expansion. Clinical/environmental statistics count all cluster members, with
unknown sources reported separately, rather than only the sketched representatives.
Reports can also read these versioned TSVs from older packages, including
IsoScout `databases-v2`, when `CLUSTER_MEMBERS` is absent. Older databases with
neither source show unavailable cluster statistics. Accession and custom builds
have no NCBI cluster membership snapshot.

## Building a database

Mashpit supports three database types:

- **Taxon database:** Build from NCBI Pathogen Detection SNP clusters for a selected organism.
- **Accession database:** Build from a user-provided list of NCBI BioSample accessions.
- **Custom database:** Build directly from local FASTA files, with no NCBI download at all.

### Taxon database

```bash
mashpit build taxon salmonella --species Salmonella
```

A specific NCBI Pathogen Detection release can be selected with `--pd_version`:

```bash
mashpit build taxon listeria_innocua \
  --species Listeria_innocua \
  --pd_version PDG000000091.9
```

Supported taxon directory names are listed in the [NCBI Pathogen Detection results directory](https://ftp.ncbi.nlm.nih.gov/pathogen/Results/).

### Accession database

Prepare a text file containing one BioSample accession per line:

```text
SAMN00000001
SAMN00000002
SAMN00000003
```

Then run (an Entrez email is required to resolve BioSample accessions via NCBI):

```bash
mashpit build accession custom_database --list biosamples.txt --email you@example.com
```

### Custom database

Build a database directly from local FASTA files - no NCBI download, Entrez lookup, or `datasets` CLI is required for this database type, which makes it suitable for proprietary or unpublished sequence collections. Place one genome per file, named `<sample_id>.fasta` (`.fa`/`.fna`/`.fas`/`.ffn`, optionally gzipped), in a directory:

```text
my_genomes/
├── sample1.fasta
├── sample2.fasta
└── sample3.fasta
```

Then run:

```bash
mashpit build custom my_database --input-dir my_genomes/
```

Optional metadata can be supplied as a TSV with a `sample_id` column (matching the filenames) plus any subset of the usual metadata fields (`strain`, `collection_date`, `geo_loc_name`, `host`, etc.) and/or your own custom columns; fields not supplied, or samples not listed, are recorded as `missing`:

```text
sample_id   strain    collection_date   host
sample1     StrainA   2024-03-01        Homo sapiens
```

```bash
mashpit build custom my_database --input-dir my_genomes/ --metadata metadata.tsv
```

A custom database has no SNP-cluster concept, so it behaves like an accession database at query time: no cluster-candidates table, and `--radius`/tree-based representative selection do not apply (every provided genome becomes a database entry directly).

### Tree-radius option

Mashpit 1.0 can optionally use a tree-radius threshold during representative selection, via `--radius` (default `20`, in the same units as the SNP tree's branch lengths). A smaller radius generally selects more representatives and gives denser coverage of within-cluster diversity. A larger radius selects fewer representatives and produces a smaller database:

```bash
mashpit build taxon salmonella --species Salmonella --radius 10
```

Use the radius option only after evaluating an appropriate value for the organism and intended surveillance resolution.

### Assembly download handling

Before representative selection and download, Mashpit removes rows without valid assembly accessions. Selected assemblies that fail to download are retried automatically. After the configured retry limit, unavailable assemblies are removed from the representative set and recorded in the build log.

This prevents missing assemblies from producing invalid database entries.

## Annotating a database

`mashpit annotate` adds or updates custom metadata columns on a database you've already built - taxon, accession, or custom - since all three share the same underlying metadata table. This is the place to attach your own project-specific fields (an internal lab ID, a project or batch code, an outbreak investigation code, anything not in Mashpit's built-in schema) without rebuilding the database.

To see which assembly/BioSample accessions are actually in the database - the ids your values file needs to match - print them as a starting point:

```bash
mashpit annotate my_database --list-accessions > accessions.tsv
```

Then annotate:

```bash
mashpit annotate my_database --values project_metadata.tsv
```

`project_metadata.tsv` needs an `asm_acc` or `biosample_acc` column to match existing rows, plus whatever columns you want to add or update:

```text
asm_acc            project_batch   internal_lab_id
GCA_000000001.1    Batch01         LAB-001
```

Columns not already present in the database are added automatically. Rows in the database not mentioned in your file keep their existing values (or are recorded as `missing` if the column is new); ids in your file with no matching database row are reported, not treated as an error.

Every `mashpit annotate` run is recorded in the database itself, so you always have a record of what changed and when:

```bash
mashpit annotate my_database --history
```

This works on databases built with older Mashpit versions too - the history table is created automatically the first time you annotate a database that doesn't have one yet, so nothing needs to be rebuilt.

## Sharding a database for faster loading

Databases are automatically split into multiple signature shards (a `DATABASE_NAME.sig/` directory of shard files instead of a single `DATABASE_NAME.sig` file) once they reach around 200 representatives, so `mashpit query` can load and compare shards across CPU cores in parallel rather than parsing one large file single-threaded. The shard count scales with database size (roughly one shard per 100 signatures, capped at 128 - so databases under ~200 representatives stay a single file, since one shard is indistinguishable from none) and query-time parallelism scales with whatever machine actually runs the query (all available cores by default, or `--threads` to cap it) - a database built once and distributed to others gets the same proportional speedup whether it's queried on a 4-core laptop or a many-core server.

Databases built with an older Mashpit version, or small databases that stayed as a single file, can be resharded without rebuilding - a one-time, local, offline operation that only rewrites the `.sig` storage, leaving all metadata untouched:

```bash
mashpit reshard my_database
```

Pass `--shards` to override the automatic count, or `--shards 1` to merge shards back into a single file:

```bash
mashpit reshard my_database --shards 64
mashpit reshard my_database --shards 1
```

## Merging compatible databases

Merge two or more local database snapshots into a **new** directory:

```bash
mashpit merge combined database_a database_b
```

Inputs can be database directories or `.db` files. Inputs remain unchanged and
an existing output is refused. The command works offline and loads the input
sketches and metadata into memory; allow enough RAM for their combined size.
The output supports the usual query, annotate, and reshard commands.

Supported combinations and requirements:

- **Custom + custom:** matching sketch settings. Identical IDs with identical
  sketches are deduplicated. Different IDs remain separate even if their sketches
  match. An ID with different sketches is rejected.
- **Taxon + taxon:** additionally require the same species, PDG release,
  representative method, and tree radius. Full cluster membership must be
  available for every represented cluster. Representatives are unioned and
  membership is deduplicated by target accession. Cluster summaries are
  recalculated from membership, never by adding input cluster-size totals.
- Mixed types, accession databases, different PDG releases, and older taxon
  databases lacking the required settings or membership are rejected. Merging
  is not a release-update or representative-reselection operation.

Actual signature settings must agree across all inputs: k-mer size, molecule
type, seed, hash count, scaled setting, and abundance tracking. Hash counts and
k-mer sizes must also agree with database descriptions. No automatic downsampling
or re-sketching takes place.

For independent custom datasets with overlapping sample IDs, explicitly prefix
both assembly and BioSample IDs, with one prefix per input:

```bash
mashpit merge combined lab_a lab_b --prefixes labA_ labB_
```

Metadata columns are unioned and missing values can be filled from another
source. Conflicting nonempty values stop the merge; no input takes priority.
Assembly IDs and BioSample IDs must each remain unique. Prefixes are available
only for custom inputs. Without prefixes, resolve conflicting metadata in a
separate copy of an input before retrying.

Taxon merges also union optional IsoScout `REP_CLUSTER` annotations by the exact,
versioned `PDS_acc`, retaining all annotation columns. Conflicting nonempty labels
or evidence fields reject the merge; a source without annotations does not erase
another source's labels. Unknown SQLite tables are rejected to prevent silent
loss of extensions.

To save the first conflicting record and field as JSON:

```bash
mashpit merge combined database_a database_b --conflict-report conflicts.json
```

The report path must be new. Compatibility errors are reported directly.
No output database is published on failure.

`merge_provenance.json` records source paths, original settings, ID mappings,
and deduplication counts. Source build tables, unavailable-record tables, and prior merge manifests
are retained under `merge_sources/`, and annotation history is preserved with source paths.
Original `database.json` and `rep_cluster_annotations.json` sidecars, when present,
are also archived there. They describe the inputs, not the merged database.
Generate a new IsoScout package manifest and checksums before using the merged
output as an IsoScout package; source manifests must not be reused for it.
Available versioned release TSVs are combined without dropping their columns;
conflicting records are rejected. Taxon output includes rebuilt
`representatives.tsv` and `cluster_summary.tsv`. Source-specific download
attempts and original FASTA paths remain in provenance rather than being
presented as properties of the merged representatives.

## Querying a database

Run a query using an assembled genome in FASTA format:

```bash
mashpit query sample.fasta path/to/database
```

Common query options include:

```text
--number                 maximum number of database matches returned
--threshold              minimum similarity used when constructing the local result tree
--annotation             metadata field used to annotate tree tips
--no-tree                skip all optional tree work; keep identical search tables
--tie-tolerance-hashes   sketch-hash tolerance used to flag near-top SNP clusters (taxon databases only)
--threads                worker processes for a sharded database (default: all available cores)
```

Example:

```bash
mashpit query sample.fasta path/to/salmonella \
  --number 200 \
  --threshold 0.85 \
  --annotation isolation_source
```

## Query output

A query produces:

1. **Representative-matches table**

   A CSV file (`<query>_representative_matches.csv`) containing the closest representatives and associated metadata. Results include fields such as BioSample accession, assembly accession, SNP-cluster accession, similarity score, and NCBI links.

2. **Cluster-candidates table** (taxon databases only)

   A CSV file (`<query>_cluster_candidates.csv`) that groups the representative matches by SNP cluster, ranked by best similarity score. A `near_top` column flags clusters whose best hit is within `--tie-tolerance-hashes` sketch hashes of the single best hit overall, since at typical sketch sizes a small similarity gap can be within normal MinHash sampling noise rather than a meaningful difference.

3. **Local tree**

   Newick and image files (`<query>_tree.newick`, `<query>_tree.png`, `<query>_tree.svg`) showing the query genome together with its closest database representatives above the similarity threshold. The tree is skipped (with a log message) when the top hit falls below `--threshold` or fewer than two candidates qualify.

4. **Tree status**

   `<query>_tree_status.json` is written after both required search tables (where applicable) succeed, even when the tree is disabled or unavailable. Its `schema_version` is `1`; `search_complete` is `true`. The `status` field is one of:

   | Status | Meaning |
   | --- | --- |
   | `generated` | Newick, PNG, and SVG were successfully written. |
   | `disabled` | The user supplied `--no-tree`. |
   | `skipped_insufficient_hits` | Fewer than two representatives qualify, including when the top hit is below the threshold. |
   | `construction_failed` | Tree-specific signature retrieval, pairwise distances, construction, annotation, or Newick writing failed. |
   | `rendering_failed` | Newick was saved, but rendering or image writing failed. |

   `reason` explains a skip/failure, and `error_type` identifies the exception for failures. `newick`, `png`, and `svg` contain filenames for successfully written outputs, otherwise `null`. A rendering failure retains valid Newick and removes partial PNG/SVG files. Previous tree artifacts with the same query prefix are removed before the optional workflow. Files that cannot be removed are listed in `cleanup_errors`; use the status fields rather than file existence to decide which artifacts are valid.

5. **Standalone HTML report**

   `<query>_report.html` opens directly in a browser, with no Streamlit or web
   server required. It includes ranked SNP clusters, full cluster sizes,
   clinical/environmental/unknown counts and percentages, sampling dates,
   computed serotypes, and representative evidence. The tree image (when
   available) and downloadable CSV/Newick files are embedded, so the report
   can be shared as one file and viewed offline. `--no-tree` still produces
   the report, with the tree marked as disabled. Accession/custom databases
   show returned-isolate metadata without SNP cluster statistics.

   The output path is printed in the query log. If report generation fails,
   Mashpit logs a warning and retains the successful CSV results.

6. **Log file**

   A timestamped record of the query parameters and processing steps.

A zero process exit code means the required search outputs succeeded; it does **not** promise a tree. Check the exit code first, then read the tree status to decide whether to display images, offer Newick alone, or show the reason. Database loading, sketching, similarity calculation, and required CSV-writing failures remain nonzero exits, even if an earlier CSV already exists. Use a fresh output directory for each query to avoid confusing earlier outputs with a failed run. If filesystem errors prevent writing the status JSON itself, Mashpit warns in the log and keeps the successful search outputs; callers must treat missing status as unavailable tree information.

For search-only use:

```bash
mashpit query sample.fasta path/to/database --no-tree
```

This skips tree-specific signature retrieval, candidate-to-candidate distances, neighbor joining, annotation, and rendering. Query-to-database comparison still runs normally. Similarity scores, ranking, tie flags, returned candidate counts, and cluster tables are identical with or without the option. The default still attempts to generate a tree.

Tree rendering and annotation use iterative traversal and serialization, avoiding TreeViz's recursive copying and BioPython's recursive traversal/writer helpers. No recursion-limit increase is used. Large trees can still exhaust memory or fail in a dependency; those optional failures are recorded without invalidating search results. Zero-distance trees use unit edges only in the display, with a caption; their Newick distances are preserved.

Mashpit results are intended for rapid screening and prioritization. They do not replace validated SNP-pipeline, cgMLST, or whole-genome phylogenetic analyses when formal outbreak confirmation is required.

## Graphical interface

Start the local Mashpit interface with:

```bash
mashpit gui
```

This launches a Streamlit app and opens it in your browser, by default at:

```text
http://localhost:8501
```

Use `--port` to run on a different port:

```bash
mashpit gui --port 8888
```

The interface includes a **Skip tree generation** setting and a downloadable tree-status JSON. Tree and preview errors leave the search tables accessible, including a Newick download when rendering failed.

The interface lets users select a local database, upload a query assembly, run the search, and inspect the result table and tree - all without the genome or results leaving the local machine.

A pre-built Mashpit database is required.

The report has three views:

- **Cluster context:** ranked full cluster sizes with environmental/clinical/unknown composition; select a cluster to explore computed serotypes, reported serovars, sampling years, geography, isolation sources, hosts, and all member isolates.
- **Sketch-distance tree:** height adapts to tip count, font size, and wrapped labels. Font and spacing controls redraw the figure without rerunning the query. The scrollable SVG view keeps labels readable; download PNG, SVG, or Newick.
- **Isolate evidence & downloads:** representative-level similarity scores, complete metadata, and the enriched cluster-summary CSV.

New taxon builds retain `CLUSTER_MEMBERS` from the complete release membership, including members without assemblies or downloaded genomes. Cluster size counts unique Pathogen Detection target accessions, not representatives. `epi_type` supplies environmental/clinical categories; unknown values remain explicit. Date summaries use valid ISO years, months, or dates and report their coverage. Raw dates, including ranges, remain in the isolate download. Computed serotype predictions are kept separate from submitted `serovar` values, and the original `computed_types` field is retained.

**Existing databases:** older taxon databases with bundled versioned membership
and metadata TSVs (including IsoScout `databases-v2`) support full cluster
statistics without rebuilding. Databases containing only representative
metadata show those statistics as unavailable until rebuilt. The report never
substitutes representative counts for full cluster size. Accession/custom
databases show metadata for returned isolates without implying SNP cluster membership.

The near-top flag uses the configured sketch-hash tolerance; it is a screening heuristic, not a statistical confidence interval. Cluster-member metadata are context: similarity was measured only for the returned representatives.

## Intended use

Mashpit is intended for fast, lightweight genomic screening, including:

- preliminary placement of newly sequenced isolates;
- identification of the most relevant NCBI Pathogen Detection SNP clusters;
- local analysis of sensitive genomic data;
- decentralized surveillance in laboratories with limited computing infrastructure;
- prioritization of isolates for higher-resolution analysis.

Mashpit should not be used as the sole evidence for declaring or excluding an outbreak relationship.

## Citation

Xu T. et al. (2024). Mashpit: sketching out genomic epidemiology.  
*Journal of Open Source Software*, 9(104), 7306.  
https://doi.org/10.21105/joss.07306

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for development and contribution guidelines.

## License

Mashpit is distributed under the GNU General Public License v2.0. See [LICENSE](LICENSE).
