Metadata-Version: 2.4
Name: cs-hashindex
Version: 20261016
Summary: A command and utility functions for making listings of file content hashcodes and manipulating directory trees based on such a hash index.
Keywords: python3
Author-email: Cameron Simpson <cs@cskk.id.au>
Description-Content-Type: text/markdown
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Requires-Dist: blake3
Requires-Dist: cs.cmdutils>=20240211
Requires-Dist: cs.context>=20250528
Requires-Dist: cs.deco>=20261016
Requires-Dist: cs.fs>=20261016
Requires-Dist: cs.fstags>=20260912
Requires-Dist: cs.hashutils>=20250414.1
Requires-Dist: cs.lex>=20261016
Requires-Dist: cs.logutils>=20260912
Requires-Dist: cs.pfx>=20260912
Requires-Dist: cs.psutils>=20260531
Requires-Dist: cs.resources>=20250915
Requires-Dist: cs.upd>=20260912
Requires-Dist: icontract
Requires-Dist: typeguard
Project-URL: MonoRepo Commits, https://bitbucket.org/cameron_simpson/css/commits/branch/main
Project-URL: Monorepo Git Mirror, https://github.com/cameron-simpson/css
Project-URL: Monorepo Hg/Mercurial Mirror, https://hg.sr.ht/~cameron-simpson/css
Project-URL: Source, https://github.com/cameron-simpson/css/blob/main/lib/python/cs/hashindex.py

A command and utility functions for making listings of file content hashcodes
and manipulating directory trees based on such a hash index.

This largely exists to solve my "what has changed remotely?" or
"what has been filed where?" problems by comparing file trees
using the files' content hashcodes.

This does require reading every file once to compute its hashcode,
but the hashcodes (and file sizes and mtimes when read) are
stored beside the file in `.fstags` files (see the `cs.fstags`
module), so that a file does not need to be reread on subsequent
comparisons.

`hashindex` knows how to invoke itself remotely using `ssh`
(this does require `hashindex` to be installed on the remote host)
and can thus be used to compare a local and remote tree, for example:

    hashindex comm -1 localtree remotehost:remotetree

When you point `hashindex` at a remote tree, it uses `ssh` to
run `hashindex` on the remote host, so all the content hashing
is done locally to the remote host instead of copying files
over the network.

You can also use it to rearrange a tree based on the locations
of corresponding files in another tree. Consider a media tree
replicated between 2 hosts. If the source tree gets rearranged,
the destination can be equivalently rearranged without copying
the files, for example:

    hashindex rearrange sourcehost:sourcetree localtree

If `fstags mv` was used to do the original rearrangement then
the hashcodes will be copied to the new locations, saving a
rescan of the source file. I keep a shell alias `mv="fstags mv"`
so this is routine for me.

A common "backup to remote" use case of mine is addressed by:

    hashindex rsync src dst

which rearranges `dst` based on `src`, then uses rsync(1) to update `dst`.

I have a backup script [`histbackup`](https://hg.sr.ht/~cameron-simpson/css/browse/bin/histbackup)
which works by making a hard link tree of the previous backup
and `rsync`ing into it.  It has long been subject to huge
transfers if the source tree gets rearranged. Now it has a
`--hashindex` option to get it to run a `hashindex rearrange`
between the hard linking to the new backup tree and the `rsync`
from the source to the new tree.

If network bandwith is limited or quotaed, you can use the
comparison function to prepare a list of files missing from the
remote location and copy them to a transfer drive for carrying
to the remote site when opportune. Example:

    hashindex comm -1 -o '{fspath}' src rhost:dst \
    | rsync -a --files-from=- src/ xferdir/

I've got a script [`pref-xfer`](https://hg.sr.ht/~cameron-simpson/css/browse/bin-cs/prep-xfer)
which does this with some conveniences and sanity checks.



Short summary:


* `dir_filepaths`: Generator yielding the filesystem paths of the files in `dirpath`.


* `dir_remap`: Generator yielding `(srcpath,[remapped_paths])` 2-tuples based on the hashcodes keying `fspaths_by_hashcode`.


* `file_checksum`: Return the hashcode for the contents of the file at `fspath`. Warn and return `None` on `OSError`.


* `hashindex`: Generator yielding `(hashcode,filepath)` 2-tuples for the files in `src`, which may be a file or a `RemotePath` or a `(host,fspath)` 2-tuple or a filesystem path. Note that this yields `(None,filepath)` for files which cannot be accessed.


* `hashindex_map`: Construct a mapping of hashcodes to filesystem paths by walking `dirpath`.


* `HashIndexCommand`: A tool to generate and use indices of file content hashcodes.


* `localpath`: Return a filesystem path modified so that it connot be misinterpreted as a remote path such as `user@host:path`.


* `main`: Commandline implementation.


* `merge`: Merge `srcpath` to `dstpath`, _preserving `dstpath` if present_. Return `True` if something was done, `False` if this was a no-op. Raise `FileExistsError` if `dstpath` exists with different content.


* `paths_remap`: Generator yielding `(srcpath,fspaths)` 2-tuples.


* `read_hashindex`: A generator which reads line from the file `f` and yields `(hashcode,fspath)` 2-tuples for each line. If there are parse errors the `hashcode` or `fspath` may be `None`.


* `read_remote_hashindex`: A generator which reads a hashindex of a remote directory, This runs: `hashindex ls -h hashname -r rdirpath` on the remote host. It yields `(hashcode,fspath)` 2-tuples.


* `rearrange`: Rearrange the files in `dirpath` according to the hashcode->[relpaths] `fspaths_by_hashcode`.


* `remote_rearrange`: Rearrange a remote directory `srcdir` on `rhost` into `dstdir` on `rhost` according to the hashcode mapping `fspaths_by_hashcode`.


* `run_remote_hashindex`: Run a remote `hashindex` command. Return the `CompletedProcess` result or `None` if `doit` is false. Note that as with `cs.psutils.run`, the arguments are resolved via `cs.psutils.prep_argv`.

# Functions

## dir_filepaths(*a, fstags: Optional[cs.fstags.FSTags] = <function uses_fstags.<locals>.<lambda> at 0x10c332c00>, **kw)

Generator yielding the filesystem paths of the files in `dirpath`.

## dir_remap(srcdirpath: str, fspaths_by_hashcode: Mapping[cs.hashutils.BaseHashCode, List[str]], *, hashname: str)

Generator yielding `(srcpath,[remapped_paths])` 2-tuples
based on the hashcodes keying `fspaths_by_hashcode`.

## file_checksum(*a, fstags: Optional[cs.fstags.FSTags] = <function uses_fstags.<locals>.<lambda> at 0x10c331da0>, **kw)

Return the hashcode for the contents of the file at `fspath`.
Warn and return `None` on `OSError`.

## hashindex(src: Union[io.TextIOBase, cs.fs.RemotePath, str, Tuple[Optional[str], str]], *, hashname: str, relative: bool = False, runstate: Optional[cs.resources.RunState] = <function uses_runstate.<locals>.<lambda> at 0x10c332520>, **kw) -> Iterable[Tuple[Optional[cs.hashutils.BaseHashCode], Optional[str]]]

Generator yielding `(hashcode,filepath)` 2-tuples
for the files in `src`, which may be a file or a `RemotePath`
or a `(host,fspath)` 2-tuple or a filesystem path.
Note that this yields `(None,filepath)` for files which cannot be accessed.

## hashindex_map(dirpath: str, *, hashname: str, relative=False) -> dict[cs.hashutils.BaseHashCode, list[str]]

Construct a mapping of hashcodes to filesystem paths
by walking `dirpath`.

## localpath(fspath: str) -> str

Return a filesystem path modified so that it connot be
misinterpreted as a remote path such as `user@host:path`.

If `fspath` contains no colon (`:`) or is an absolute path
or starts with `./` then it is returned unchanged.
Otherwise a leading `./` is prepended.

## main(argv=None)

Commandline implementation.

## merge(*a, fstags: Optional[cs.fstags.FSTags] = <function uses_fstags.<locals>.<lambda> at 0x10c333b00>, **kw)

Merge `srcpath` to `dstpath`, _preserving `dstpath` if present_.
Return `True` if something was done, `False` if this was a no-op.
Raise `FileExistsError` if `dstpath` exists with different content.

This is aimed at situations such as merging downloads with
an existing corpus, which might have hard links etc, so
`dstpath` is the important half of the pair.

NB: `symlink_mode` is currently disabled.

If 'dstpath' exists, checksum their contents and raise
`FileExistsError` if they differ.
If `dstpath` does not exist, move/link/symlink `srcpath` to `dstpath`.
This also merges the fstags from `srcpath` to `dstpath`.

Otherwise the files have the same content, merge while preserving `dstpath`.

## paths_remap(srcpaths: Iterable[str], fspaths_by_hashcode: Mapping[cs.hashutils.BaseHashCode, List[str]], *, hashname: str)

Generator yielding `(srcpath,fspaths)` 2-tuples.

## read_hashindex(f, start=1, *, hashname: str) -> Iterable[Tuple[Optional[cs.hashutils.BaseHashCode], Optional[str]]]

A generator which reads line from the file `f`
and yields `(hashcode,fspath)` 2-tuples for each line.
If there are parse errors the `hashcode` or `fspath` may be `None`.

## read_remote_hashindex(rhost: str, rdirpath: str, *, hashname: str, quiet=True, ssh_exe: str, hashindex_exe: str, relative: bool = False) -> Iterable[Tuple[Optional[cs.hashutils.BaseHashCode], Optional[str]]]

A generator which reads a hashindex of a remote directory,
This runs: `hashindex ls -h hashname -r rdirpath` on the remote host.
It yields `(hashcode,fspath)` 2-tuples.

Parameters:
* `rhost`: the remote host, or `user@host`
* `rdirpath`: the remote directory path
* `hashname`: the file content hash algorithm name
* `ssh_exe`: optional `ssh` command
* `hashindex_exe`: the remote `hashindex` executable
* `relative`: optional flag, default `False`;
  if true pass `'-r'` to the remote `hashindex ls` command
* `check`: whether to check that the remote command has a `0` return code,
  default `True`

## rearrange(srcdirpath: str, rfspaths_by_hashcode, dstdirpath: str | None = None, *, hashname: str, move_mode: bool = False, once: bool = False, symlink_mode=False, do0=False, doit: bool, fstags: cs.fstags.FSTags, runstate: Optional[cs.resources.RunState] = <function uses_runstate.<locals>.<lambda> at 0x10c3332e0>, quiet: bool)

Rearrange the files in `dirpath` according to the
hashcode->[relpaths] `fspaths_by_hashcode`.

Parameters:
* `srcdirpath`: the directory whose files are to be rearranged
* `rfspaths_by_hashcode`: a mapping of hashcode to relative
  pathname to which the original file is to be moved
* `dstdirpath`: optional target directory for the rearranged files;
  defaults to `srcdirpath`, rearranging the files in place
* `hashname`: the file content hash algorithm name
* `move_mode`: move files instead of linking them
* `symlink_mode`: symlink files instead of linking them
* `do0`: rearrange files even of they are empty
* `doit`: if true do the link/move/symlink, otherwise just print

## remote_rearrange(rhost: str, srcdir: str, dstdir: str, fspaths_by_hashcode: Mapping[cs.hashutils.BaseHashCode, List[str]], *, doit: bool, hashindex_exe: str, hashname: str, move_mode: bool, once: bool, quiet: bool, symlink_mode: bool)

Rearrange a remote directory `srcdir` on `rhost` into `dstdir`
on `rhost` according to the hashcode mapping `fspaths_by_hashcode`.

## run_remote_hashindex(rhost: str, argv, *, hashindex_exe: str, **subp_options)

Run a remote `hashindex` command.
Return the `CompletedProcess` result or `None` if `doit` is false.
Note that as with `cs.psutils.run`, the arguments are resolved
via `cs.psutils.prep_argv`.

Parameters:
* `rhost`: the remote host, or `user@host`
* `argv`: the command line arguments to be passed to the
  remote `hashindex` command
* `check`: whether to check that the remote command has a `0` return code,
  default `True`
Other keyword parameters are passed therough to `cs.psutils.run`.

# Classes

## class HashIndexCommand(cs.cmdutils.BaseCommand)

A tool to generate and use indices of file content hashcodes.

Usage summary:

    Usage: hashindex [common-options...] subcommand [options...]
      A tool to generate and use indices of file content hashcodes.
      Subcommands:
        comm [common-options...] {-1|-2|-3|-r} {path1|-} {path2|-}
          Compare the filepaths in path1 and path2 by content.
          Options:
            -1  List hashes and paths only present in path1.
            -2  List hashes and paths only present in path2.
            -3  List hashes and paths present in path1 and path2.
            -r  Emit relative paths in the listing.
        help [common-options...] [-l] [-s] [subcommand-names...]
          Print help for subcommands.
          This outputs the full help for the named subcommands,
          or the short help for all subcommands if no names are specified.
          Options:
            -l  Long listing.
            -r  Recurse into subcommands.
            -s  Short listing.
        info [common-options...] [field-names...]
          Recite general information.
          Explicit field names may be provided to override the default listing.
        ls [common-options...] [options...] [[host:]path...]
          Walk filesystem paths and emit a listing.
          The default path is the current directory.
          In quiet mode (-q) the hash indicies are just updated
          and nothing is printed.
          Options:
            -r  Emit relative paths in the listing.
                This requires each command line path to be a directory.
        rearrange [common-options...] [--refdir {[[user@]host:]refdir|-}] [[user@]rhost:]srcdir [dstdir]
          Rearrange files from srcdir into dstdir based on their positions in refdir.
          Arguments:
            refdir    The reference directory, which may be local or remote
                      or "-" indicating that a hash index will be read from
                      standard input.
            srcdir    The directory containing the files to be rearranged,
                      which may be local or remote.
            dstdir    Optional destination directory for the rearranged files.
                      Default is the srcdir.
          Examples:
            rearrange [common-options...] backupdir
              Rearrange the contents of backupdir based on the current directory.
            rearrange [common-options...] --refdir ~/media remote:media
              Rearrange the remote media directory based on the local media directory.
            rearrange [common-options...] remote:dldir remote:storedir
              Rearrange the files in a remote download directory
              into a remote storage directory based on the locations
              in the local current directory.
          Options:
            -1               Rearrange only one file.
            --ln             Hard link files instead of moving them.
            --refdir refdir  Reference directory, default the current directory.
            -s               Symlink mode.
        repl [common-options...]
          Run a REPL (Read Evaluate Print Loop), an interactive Python prompt.
          Options:
            --banner banner  Banner.
        rsync [common-options...] [options] srcdir dstdir [rsync-options...]
          Rearrange dstdir according to srcdir then rsync srcdir into dstdir.
          Options:
            --bwlimit bwlimit  Rsync bandwidth limit, passed to rsync.
            --delete           Delete from dstdir, passed to rsync.
            --partial          Keep partially transferred files, passed to rsync.
        shell [common-options...]
          Run a command prompt via cmd.Cmd using this command's subcommands.

### `HashIndexCommand.Options(cmd: Optional[str] = None, dry_run: bool = False, force: bool = False, verbosity: int = 0, runstate: Optional[cs.resources.RunState] = None, runstate_signals: Tuple[int] = (<Signals.SIGHUP: 1>, <Signals.SIGINT: 2>, <Signals.SIGQUIT: 3>, <Signals.SIGTERM: 15>), ssh_exe: str = <factory>, hashname: str = 'blake3', move_mode: bool = False, hashindex_exe: str = 'hashindex', symlink_mode: bool = False, relative: Optional[bool] = None, output_format: str = '{hashcode} {fspath}') -> None`

Options for `HashIndexCommand`.

### `HashIndexCommand.SUBCOMMAND_ARGV_DEFAULT`

    'ls'

### `HashIndexCommand.USAGE_FORMAT`

    ('Usage: {cmd} subcommand [options...]\n'
     '  A tool to generate and use indices of file content hashcodes.')

### `HashIndexCommand.USAGE_KEYWORDS`

    {'HASHNAME_DEFAULT': 'blake3', 'OUTPUT_FORMAT_DEFAULT': '{hashcode} {fspath}'}

### `HashIndexCommand.cmd_comm(self, argv, *, runstate: Optional[cs.resources.RunState] = <function uses_runstate.<locals>.<lambda> at 0x10c331440>)`

Usage: {cmd} {{-1|-2|-3|-r}} {{path1|-}} {{path2|-}}
Compare the filepaths in path1 and path2 by content.
Options:
  -1  List hashes and paths only present in path1.
  -2  List hashes and paths only present in path2.
  -3  List hashes and paths present in path1 and path2.
  -r  Emit relative paths in the listing.

### `HashIndexCommand.cmd_ls(self, argv, *, runstate: Optional[cs.resources.RunState] = <function uses_runstate.<locals>.<lambda> at 0x10c331940>)`

Usage: {cmd} [options...] [[host:]path...]
Walk filesystem paths and emit a listing.
The default path is the current directory.
In quiet mode (-q) the hash indicies are just updated
and nothing is printed.
Options:
  -r  Emit relative paths in the listing.
      This requires each command line path to be a directory.

### `HashIndexCommand.cmd_rearrange(self, argv)`

Usage: {cmd} [--refdir {{[[user@]host:]refdir|-}}] [[user@]rhost:]srcdir [dstdir]
Rearrange files from srcdir into dstdir based on their positions in refdir.
Arguments:
  refdir    The reference directory, which may be local or remote
            or "-" indicating that a hash index will be read from
            standard input.
  srcdir    The directory containing the files to be rearranged,
            which may be local or remote.
  dstdir    Optional destination directory for the rearranged files.
            Default is the srcdir.
Examples:
  {cmd} backupdir
    Rearrange the contents of backupdir based on the current directory.
  {cmd} --refdir ~/media remote:media
    Rearrange the remote media directory based on the local media directory.
  {cmd} remote:dldir remote:storedir
    Rearrange the files in a remote download directory
    into a remote storage directory based on the locations
    in the local current directory.
Options:
  -1               Rearrange only one file.
  --ln             Hard link files instead of moving them.
  --refdir refdir  Reference directory, default the current directory.
  -s               Symlink mode.

### `HashIndexCommand.cmd_rsync(*a, fstags: Optional[cs.fstags.FSTags] = <function uses_fstags.<locals>.<lambda> at 0x10c331800>, **kw)`

Usage: {cmd} [options] srcdir dstdir [rsync-options...]
Rearrange dstdir according to srcdir then rsync srcdir into dstdir.
Options:
  --bwlimit bwlimit  Rsync bandwidth limit, passed to rsync.
  --delete           Delete from dstdir, passed to rsync.
  --partial          Keep partially transferred files, passed to rsync.

### `HashIndexCommand.poppathspec(argv: List[str], name: str = 'dirspec', check_isdir=False) -> cs.fs.RemotePath`

Pop a leading dirspec from `argv`, a filesystem path with
an optional leading `[user@]rhost:` prefix.
Return a `(host,fspath)` 2-tuple being the remote host (`None` if omitted)
and the filesystem path.
Raises `GetoptError` on a missing or invalid argument.

### `HashIndexCommand.run_context(*a, fstags: Optional[cs.fstags.FSTags] = <function uses_fstags.<locals>.<lambda> at 0x10c330e00>, **kw)`

Sanity check the hashname, open the fstags.
