Metadata-Version: 2.4
Name: WhiskiWrap
Version: 1.2.2
Summary: Whisk package wrapper created by cxrodgers
Home-page: http://pypi.python.org/pypi/WhiskiWrap/
Author: Chris Rodgers, Ariel Iporre, Vincent Prevosto
Author-email: 
Maintainer: vncntprvst
License: The Janelia Farm Research Campus Software Copyright 1.1
        
        Copyright (c) 2010, Howard Hughes Medical Institute, All rights reserved.
        
        Redistribution and use in source and binary forms, with or without
        modification, are permitted provided that the following conditions are met:
        
        1. Redistributions of source code must retain the above copyright notice, this
        list of conditions and the following disclaimer.
        
        2. Redistributions in binary form must reproduce the above copyright notice,
        this list of conditions and the following disclaimer in the documentation
        and/or other materials provided with the distribution.
        
        3. Neither the name of the Howard Hughes Medical Institute nor the names of its
        contributors may be used to endorse or promote products derived from this
        software without specific prior written permission.
        
        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
        ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, ANY IMPLIED
        WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT, OR FITNESS FOR A PARTICULAR
        PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS
        BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
        CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
        GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; REASONABLE ROYALTIES; OR
        BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
        IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
        ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
        POSSIBILITY OF SUCH DAMAGE.
        
Project-URL: Homepage, http://pypi.python.org/pypi/WhiskiWrap/
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: tables>=3.5.1
Requires-Dist: pandas
Requires-Dist: MediaInfo
Requires-Dist: future
Requires-Dist: tifffile
Requires-Dist: imagecodecs
Requires-Dist: statsmodels
Requires-Dist: ffmpeg-python>=0.2.0
Requires-Dist: whisk-janelia[ffmpeg]
Requires-Dist: zarr
Requires-Dist: pyarrow
Requires-Dist: matplotlib
Requires-Dist: easygui
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# WhiskiWrap
WhiskiWrap provides tools for running [whisk](http://whiskertracking.janelia.org) more easily and efficiently.

My goal is to improve whisk in the following ways:

1. Make it more flexible about reading various input files. In my experience whisk has trouble reading certain input videos. Instead, WhiskiWrap uses your system's ffmpeg to read input files (because ffmpeg can typically read almost anything) and to generate simple tiff stacks which whisk can reliably read.
2. Make it faster, by calling many instances of `trace` in parallel on non-overlapping chunks of the input video.
3. Make it more cross-platform and memory-efficient, by converting whisk's output files into HDF5 files which can be read by multiple programs (Python, Matlab) on any operating system. Importantly, HDF5 files can also be read partially to avoid overflowing your system's memory.

## Example
Note: The current best pipeline for running trace is `interleaved_reading_and_trace`, but the command `pipeline_trace` is the only one configured to run measure.

First start the interactive python environment by typing ipython at the terminal.

Import WhiskiWrap so it can be used:

`import WhiskiWrap`

Set the path to the input file. There are some test videos in this repository that you can use. Usually it's best to copy this file to a new directory, because a lot of temporary files will be created in the same directory.
```
mkdir ~/whiski_wrap_session
cp ~/dev/WhiskiWrap/test_video2.mp4 ~/whiski_wrap_session/test_video2.mp4
input_video = '~/whiski_wrap_session/test_video2.mp4'
```

Choose where you want the output HDF5 file to be.

`output_file = 'output.hdf5'`

Run the trace. Here we use 4 parallel processes.

`WhiskiWrap.pipeline_trace(input_video, output_file, n_trace_processes=4)`

If you go to the session directory, you'll see a bunch of tiff stacks and whiskers files that were generated by every instance of trace. There's also a combined HDF5 file with all the data combined. You can read it into Python like so:
```
import tables
import pandas
with tables.open_file(output_file) as fi:`
  test_result = pandas.DataFrame.from_records(
    fi.root.summary.read())     
```
This just reads the "summary": the tip and follicle of every whisker in every frame. The HDF5 file also contains the x- and y-coordinates of every pixel in every whisker, but you probably don't want to read all of this in at once.

## More detail on how WhiskiWrap works
1. Split the entire video into _epochs_ of about 100K frames (~100MB of data). The entire epoch will be read into memory, so the epoch size cannot be too big.
2. For each epoch:
  1. Split it into _chunks_ of about 1000 frames, each of which will be traced separately. The frames can optionally be cropped at this point.
  2. Write each chunk to disk as a tiff stack (note: these files are quite large).
  3. Trace each chunk with parallel instances of `trace`. A `whiskers` file is generated for each chunk.
  4. Parse in order each chunk's `whiskers` file and append the results to an output HDF5 file.
  5. (Optional) delete the intermediate chunk files here.

The following parameters must be chosen:
* `n_trace_processes` - the number of parallel instances of `trace` to run at the same time. The most efficient choice is the number of CPUs on your system.
* `epoch_sz_frames` - the number of frames per epoch. It is most efficient to make this value as large as possible. However, it should not be so large that you run out of memory when reading in the entire epoch of video. 100000 is a reasonable choice.
* `chunk_sz_frames` - the size of each chunk. Ideally, this should be `epoch_size` / `n_trace_processes`, so that all the processes complete at about the same time. It could also be `epoch_size` / (N * `n_trace_processes`) where N is an integer.

You may also add optional parameters to run the measure command
* `measure=True` - run measure command, default is False
* `face='right'` - run measure with face on right side, can also specify to 'left' side

# Installation
WhiskiWrap is written in Python and relies on `ffmpeg` for reading input videos, `tifffile` for writing tiff stacks, `whisk` for tracing whiskers in the tiff stacks, and `pytables` for creating HDF5 files with all of the results.
Also make sure that you have installed `ffmpeg-python>=0.2.0`.

## Installing `ffmpeg`
First install [`ffmpeg`](https://www.ffmpeg.org/) and ensure it is available on your system path -- that is, you should be able to type `ffmpeg` in the terminal and it should find it and run it.

## Installing `whisk`
Next install [`whisk`](http://whiskertracking.janelia.org). There are several ways to do this:

1. Download the pre-built binary. This is the easiest path because it doesn't require compiling anything. However, you still need to make a few changes to the Python code that is downloaded in order to make it work with `WhiskiWrap`.
2. Build `whisk` from source, using my lightly customized fork. This will probably require more trouble-shooting to make sure all of its parts are working.

To use the pre-built binary (preferred):
1. Download the [zipped binary](http://whiskertracking.janelia.org/wiki/display/MyersLab/Whisker+Tracking+Downloads):
    1. Windows: Download `whisk-1.1.0d-win32.exe` or `whisk-1.1.0d-win64.exe` install adding to the PATH.
    2. Linux: Unpack with `tar -xzf whisk-1.1.0d-64bit-Linux.tar.gz`. Rename the unpacked directory to `[Path-to-WhiskiWrap-Cloned-Repo]/whisk`.Add to `echo 'export WHISKPATH=[WhiskiWrap-Cloned-Path]/whisk' >> ~/.bashrc`
    3. Mac: Unpack with `tar -xzf whisk-1.1.0d-64bit-Linux.tar.gz`. Rename the unpacked directory to `[Path-to-WhiskiWrap-Cloned-Repo]/whisk`. Add to `echo 'export WHISKPATH=[WhiskiWrap-Cloned-Path]/whisk' >> ~/.bash_profile`
2. Add the binaries to your system path so that you can run `trace` from the command line.
4. Install whisk package:
    1. `git clone https://github.com/aiporre/whisk.git`
    2. Copy recursively `[Path-to-WhiskiWrap-Cloned-Repo]/whisk/lib/whisk/` to `[Path-to-whisk-Cloned-Repo]/whisk/`
    3. Install whisk package `cd [Path-to-whisk-Cloned-Repo] && pip install .`
3. In the `WhiskiWrap` repo install now this package with: `uv pip install .` (or `pip install .` if not using uv)
4. Test that everything worked by opening python or ipython and running `import WhiskiWrap`

To build from source:

1. Install required dependencies (gl.h, etc)
2. Download the source from my lightly modified fork, which makes the `__init__` changes described above.
3. `cd ~/dev`
4. `git clone https://github.com/aiporre/whisk.git`
5. `cd whisk`
6. `mkdir build && cd build`
7. `cmake ..`
8. `make`
9. Copy a library into an expected location: `[Path-to-whisk-Cloned-Repo]/bin`
    1. In Windows: append to the `PATH` environmental variable `[Path-to-whisk-Cloned-Repo]/bin`
    2. In Linux:  `echo 'export WHISKPATH=[Path-to-whisk-Cloned-Repo]/bin/' >> ~/.bashrc`
    3. In MacOS:  `echo 'export WHISKPATH=[Path-to-whisk-Cloned-Repo]/bin/' >> ~/.profile_bash`
10. copy `cp libwhisk.so ../whisk` in Linux or mac, and `copy whisk.dll ..\whisk` in windows.
11. 'cd ..'
12. When you create your conda environment you should add whisk with `uv pip install .` (or `pip install .` if not using uv)
13. Test that everything worked by opening python or ipython and running `from whisk import traj, trace`
14. You need to the binaries to then environmental variables `WHISKPATH` and/or `PATH` 

## Installing Python modules

### Recommended: Using uv (Modern Python Package Manager)

The easiest and fastest way to install WhiskiWrap is using [uv](https://docs.astral.sh/uv/), a modern Python package manager. This project now includes a `pyproject.toml` file that makes installation with uv straightforward.

**Requirements**:

* Python 3.10 or higher
* Some dependencies may require compilation. If you encounter build errors:
  * On Debian/Ubuntu: `sudo apt-get install python3-dev build-essential`
  * On Fedora/RHEL: `sudo dnf install python3-devel gcc`
  * On macOS: `xcode-select --install`
  * On Windows: Install [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)

1. First, install uv if you haven't already:

   For Windows, you can download the installer from the [uv releases page](https://github.com/astral-sh/uv/releases).
   For Linux and macOS, you can use the following command to install uv:

   ```bash
   curl -LsSf https://astral.sh/uv/install.sh | sh
   ```

2. **Option A: Install in an isolated virtual environment (Recommended)**
   
   Create and activate a virtual environment, then install WhiskiWrap:

   ```bash
   uv venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   uv pip install -e .
   ```
   
   This keeps WhiskiWrap and its dependencies isolated from your system Python and other projects.

3. **Option B: Install globally (simpler but less isolated)**
   
   Install WhiskiWrap system-wide:

   ```bash
   uv pip install -e .  # Development mode
   # or
   uv pip install .     # Normal installation
   ```
   
   This makes WhiskiWrap available everywhere but may cause dependency conflicts with other projects.

The `pyproject.toml` file will automatically handle all the dependencies, including matplotlib and easygui.

**Important**: If you chose Option A (virtual environment), you'll need to activate the environment each time you want to use WhiskiWrap:

```bash
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
```

Then you can use WhiskiWrap as described in the examples above.

### Alternative: Using conda (Traditional Method)

If you prefer using conda to manage and install Python modules:

1. Create conda environment: `conda create -n WhiskiWrap python=3.10 pip` (conda automatically installs required build dependencies)
2. Activate with: `conda activate WhiskiWrap`
3. Install WhiskiWrap: `pip install .` in the directory of this project
4. Install additional packages if needed:

   ```bash
   conda install scipy matplotlib
   pip install tifffile easygui
   ```

Note: When using conda, you should not have anything on your `$PYTHONPATH`, and there shouldn't be any installed modules in your `~/.local`.
