Metadata-Version: 2.4
Name: shellhistory
Version: 0.3.0
Summary: Visualize your usage of Bash/Zsh through a web app thanks to Flask and Highcharts!
Keywords: shell,bash,zsh,history,analysis,visualization
Author-Email: Timothée Mazzucotelli <dev@pawamoy.fr>
License-Expression: ISC
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Documentation
Classifier: Topic :: Software Development
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Project-URL: Homepage, https://pawamoy.github.io/shellhistory
Project-URL: Documentation, https://pawamoy.github.io/shellhistory
Project-URL: Changelog, https://pawamoy.github.io/shellhistory/changelog
Project-URL: Repository, https://github.com/pawamoy/shellhistory
Project-URL: Issues, https://github.com/pawamoy/shellhistory/issues
Project-URL: Discussions, https://github.com/pawamoy/shellhistory/discussions
Project-URL: Gitter, https://gitter.im/shellhistory/community
Project-URL: Funding, https://github.com/sponsors/pawamoy
Requires-Python: >=3.11
Requires-Dist: flask<4,>=3.1.3
Requires-Dist: flask-admin<2.1,>=2.0.2
Requires-Dist: sqlalchemy<2.1,>=1.4
Requires-Dist: tqdm>=4.28
Description-Content-Type: text/markdown

# Shell History

[![ci](https://github.com/pawamoy/shellhistory/workflows/ci/badge.svg)](https://github.com/pawamoy/shellhistory/actions?query=workflow%3Aci)
[![documentation](https://img.shields.io/badge/docs-zensical-FF9100.svg?style=flat)](https://pawamoy.github.io/shellhistory/)
[![pypi version](https://img.shields.io/pypi/v/shellhistory.svg)](https://pypi.org/project/shellhistory/)
[![gitter](https://img.shields.io/badge/matrix-chat-4DB798.svg?style=flat)](https://app.gitter.im/#/room/#shellhistory:gitter.im)

Visualize your usage of Bash/Zsh through a web app thanks to Flask and Highcharts!

## Installation

```bash
pip install shellhistory
```

With [`uv`](https://docs.astral.sh/uv/):

pipx install --python python3.6 shellhistory
```

## Setup

`shellhistory` needs a lot of info to be able to display various charts.
The basic shell history is not enough. In order to generate the necessary
information, you have to enable the shell extension.

At shell startup, in `.bashrc` or `.zshrc`, put the following:

```bash
# only load it for interactive shells
if [[ $- == *i* ]] && command -v shellhistory-location &>/dev/null; then
    . $(shellhistory-location)
    shellhistory enable
fi
```

... and now use your shell normally!

If you want to stop `shellhistory`, simply run `shellhistory disable`.

**Note:** *for performance reasons, you can also use the static,
absolute path to the source file.
Indeed, calling `shellhistory-location` spawns a Python process
which can slow down your shell startup.
Get the path once with `shellhistory-location`, and use `. <ABS_PATH>`.
In my case it's `. ~/.local/pipx/venvs/shellhistory/lib/python3.6/site-packages/shellhistory/shellhistory.sh`.*

## Usage

Launch the web app with `shellhistory-web`.
Now go to [http://localhost:5000/](http://localhost:5000/) and enjoy!

You will need Internet connection since assets are not bundled.

### Filtering and splitting

Every chart is served by the same mechanism, so all of them take the same
query string. Pick one machine, or one terminal, or one project, and every
chart narrows to it; ask for a split, and it comes back as one series per
value of that dimension instead of one:

| Parameter | What it does |
| --- | --- |
| `split=host` | one series per machine — also `user`, `shell`, `level`, `status`, `type` |
| `split=env` | one series per place the shell was typed — also `terminal`, `editor`, `wm`, `multiplexer`, `location`, `display_manager`, `container` |
| `host=corsair` | keep only that value; works for every dimension above |
| `path=/home/me/dev/thing` | keep only the commands run under that directory |
| `cmd=git` | keep only the command lines starting with that |
| `since=2024-01-01`, `until=2024-12-31` | keep only that span |
| `granularity=day` | the bucket of the charts reported over time: `day`, `week`, `month`, `year` |
| `normalize=1` | report each series as a share of its own total, so a busy machine and a quiet one can be compared |

The same controls sit in a bar above every chart, so none of this has to be
typed by hand.

### What the environment charts know

The shell records the whole chain of processes above it, which turns out to
name the terminal emulator, the editor hosting that terminal, the window
manager, the display manager, the multiplexer, and whether an `sshd` sits in
between. None of that is stored a second time: a whole database holds only a
couple of thousand distinct ancestries, so they are read on the fly.

## Some technical info

### How it works

When you enter a command, `shellhistory` will compute values
*before* and *after* the command execution.
In Bash, it uses a trap on DEBUG and the PROMPT_COMMAND variable
(`man bash` for more information).
For Zsh, it uses the preexec_functions and precmd_functions arrays
(anyone knows where to find the official documentation for these?
Some information in `man zshmisc`).

Before the command is executed, we start a timer, compute the command type,
and store the current working directory and the command itself.

After the command has finished, we store the return code, and stop the timer.

### Storage

Records go straight into a SQLite database (`~/.shellhistory/db.sqlite3` by
default, `$SHELLHISTORY_DB` to override). At the end of each command the shell
spawns a small detached writer, `_record.py`, which uses nothing but the standard
library and never makes the prompt wait on it.

### Secret scanning and redaction

Install [Gitleaks](https://github.com/gitleaks/gitleaks), then scan recorded
commands locally. Each finding shows the recorded command, its matching rules,
and the proposed redaction. Choose **apply** to use the proposal, **redact** to
write a replacement command, **keep** to record an intentional exception, or
**skip** to leave the command and its scan state unchanged:

```console
shellhistory secrets
```

Applied and kept choices are recorded in `secret_scans`, so a reviewed command
is not returned by later scans unless you use `--rescan`; skipped commands are
returned again. Close the web app and other processes holding the database open
before reviewing findings. To redact every finding without prompting:

```console
shellhistory secrets --apply
```

Commands are sent to Gitleaks in batches (500 by default). Provider-specific
Gitleaks rules are extended with rules for shell options, authorization headers,
credentials in URLs, and common short password options. A canary in every batch
makes the scan fail rather than silently marking rows clean when Gitleaks or its
configuration is broken.

Applied scans record each row as `clean`, `kept`, or `redacted` in `secret_scans`;
commands without a record are unscanned. Later scans skip recorded rows, so use
`shellhistory secrets --rescan` (or add `--apply`) after upgrading Gitleaks or
changing scanner rules. Redaction replaces only the extracted value with
`[REDACTED]`; if a finding cannot be mapped back to an exact value, the whole
command is redacted instead.

An applied scan enables SQLite secure deletion, vacuums the database, and
truncates its write-ahead log. Copies and backups made before the scan remain
sensitive, and detected credentials should still be rotated.

The schema has three tables. Everything that stays the same for the whole life
of a shell -- host, user, tty, shell, level and the process ancestry -- is
written once into `sessions`; `history` holds what changes per command (start,
stop, type, return code, working directory, the command itself) and points at
its session; `secret_scans` optionally records the latest scan state for a
history row. The ancestry string alone is a few hundred bytes and there are
only a couple of thousand distinct ones, so repeating it on every row used to
account for half the database file.

Values are passed to the writer as separate arguments and bound as query
parameters, so nothing in a command line can be mistaken for a field separator
or for SQL. That is why no encoding is needed: newlines, colons and quotes are
all stored verbatim.

SQLite is configured for many small concurrent writers -- WAL journaling so
writers do not block readers, and a busy timeout so a shell that loses the race
waits its turn instead of dropping the record. If the database cannot be written
at all, the record is appended to `~/.shellhistory/unrecorded.jsonl` rather than
lost.

#### Legacy text format

Earlier versions appended to a colon-delimited text file, base64-encoding the
paths and ancestry to protect the delimiter, with `;` prefixing continuation
lines of a multi-line command:

```
:start:stop:uuid:parents:host:user:tty:path:shell:level:type:code:command
```

`shellhistory-cli --import [FILE]` still reads that format, so archived history
files can be loaded. A database created by an older version is converted with
`shellhistory-cli --migrate`, which keeps the original alongside as a backup.
### How we get the values

Start and stop time are obtained with `date '+%s%N'`, return code is passed
directly with `$?`, working directory is obtained with `$PWD` and command
type with `type` for Bash and `whence` for Zsh.

Values for UUID, parents, hostname, and TTY are computed only once, when
`shellhistory.sh` is sourced. Indeed they do not change during usage of the current
shell process. Hostname and TTY are obtained through commands `hostname` and
`tty`. UUID is generated with command `uuidgen`. Also note that UUID
is exported in subshells so we know which shell is a subprocess of another, and
so we are able to group shell processes by "sessions", a session being an opened
terminal (be it a tab, window, pane or else). Parents are obtained with a
function that iteratively greps `ps` result with PIDs (see `shellhistory.sh`).

Values for user, shell, and level are simply obtained through environment
variables: `$USER`, `$SHELL` (though its use here is incorrect:
see [issue 24](https://github.com/pawamoy/shellhistory/issues/24)),
and `$SHLVL` (also see [issue 25](https://github.com/pawamoy/shellhistory/issues/25)).

The last command is obtained with the command `fc`.
Using `fc` allows `shellhistory` to have the same behavior as your history:
- if commands starting with spaces are ignored, they will be ignored
  in `shellhistory` as well.
- same for duplicates (entering `ls` two or more times
  saves only the first instance). Note however that if you type the same command
  as the previous one in an other terminal, it will still be appended,
  unless you manage to synchronize your history between terminals,
  which is another story.

Additionally, if you enter an empty line,
or hit Control-C before enter, nothing will be appended either.
The trick behind this is to check the command number in the current history
(see `shellhistory.sh` for technical details).
