Metadata-Version: 2.4
Name: id_scrambler
Version: 1.0.0
Summary: Offline cleartext scrambling tool driven by YAML pattern rules.
Author: Nokia
License-Expression: BSD-3-Clause
Project-URL: Homepage, https://github.com/nokia-steward/id-scrambler/
Project-URL: Source, https://github.com/nokia-steward/id-scrambler/
Keywords: id_scrambler,anonymization,sanitize,yang,sha256
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Security
Classifier: Topic :: Utilities
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=6.0
Requires-Dist: tqdm>=4.70.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0; extra == "dev"
Dynamic: license-file

# Identifiable information scrambler for Nokia SR OS (id-scrambler)

## Overview

The Nokia `id-scrambler` application scrambles cleartext files containing
identifiable information, such as the output from the
`admin show configuration` (`admin display-config` in classic CLI) or the
`admin tech-support state` (`admin tech-support output-level state` in classic
CLI) commands. This allows an administrator to review and edit the scrambled
cleartext output before sending it to Nokia technical support, ensuring all
identifiable data has been scrambled.

The following identifiable information is scrambled:

- quoted strings that may contain descriptions or names of items such as:
  - BGP communities
  - domain names
  - interfaces
  - local users
  - policies
  - services
  - the router
  - VPRNs
- MAC addresses
- IP addresses and prefixes
- autonomous system numbers
- log events

## Scrambling

The application reads cleartext input line by line, replaces cleartext strings
that match patterns in a YAML rule file with a HMAC-SHA256 scrambled string,
and writes cleartext output.

Strings that match a pattern are scrambled with
`hash = HMAC-SHA256(key, string)` to generate a 64-character hexadecimal hash.
The resulting scrambled string consists of the `scramble-prefix`, followed by
the first eight characters of the hash, followed by the optional
`scramble-suffix`. For example, `STR-8f96d576`.

> WARNING: Review the scrambled output carefully.

The same key produces the same scrambled string for the same input. This
ensures that a cleartext string like an IP address has the same scrambled
string and can be correlated in the same output or multiple outputs without
revealing its real value.

## Requirements

- Python 3.11 or newer, tested on Python 3.11 through 3.14
- [PyYAML](https://pypi.org/project/PyYAML/) 6.0 or newer
- [tqdm](https://pypi.org/project/tqdm/) 4.70.0 or newer

The application can be downloaded from:

- [GitHub](https://github.com/nokia-steward/id-scrambler/)
- [PyPi](https://pypi.org/project/id-scrambler/)

## Installation

Nokia recommends using Python virtual environments, execute these commands to
active the virtual environment:

```sh
python3 -m venv .venv
source .venv/bin/activate
```

### Installation from GitHub

To install the application and its dependencies after cloning from GitHub,
execute this command:

```sh
pip install .
```

### Installation from PyPi

To install the application and its dependencies from PyPi, execute this
command:

```sh
pip install id-scrambler
```

To upgrade to the latest release from PyPi, execute this command:

```sh
pip install --upgrade id-scrambler
```

### Docker installation

Docker container images for x86 and ARM platforms are available on the GitHub
Container Registry.

To install the 64-bit x86 image, execute this command:

```sh
sudo docker pull ghcr.io/nokia-steward/id-scrambler:1.0.0-amd64
```

To install the 64-bit ARM image, execute this command:

```sh
sudo docker pull ghcr.io/nokia-steward/id-scrambler:1.0.0-arm64
```

## Usage

```
Usage: id_scrambler [OPTION]... [FILE]...
Scramble strings that match patterns in a cleartext file.

With no FILE, or when FILE is -, read standard input.

  -c, --config        the YAML configuration file, if not specified the file with
                      a ".yml" suffix in the current working directory is used
  -d, --debug         enable debug mode and send output to stderr
  -h, --help          display this help text and exit
  -k, --key           the key to be used in the hash, 32 to 128 bytes,
                      if not specified a random 32-byte key is generated
  -o, --output        the output file name, if not specified a ".scrm" suffix is
                      added to the input file name, if "stdout" then the output
                      is sent to stdout
  -v, --version       display version information and exit
```

## Scrambling rules and options

Scrambling rules are defined in a YAML file. When `-c/--config` is omitted,
the application uses the file with a `.yml` suffix in the current working
directory.

Each top-level key is a rule name. Rule ordering is significant: the first
rule that matches a string of text is evaluated, so ignore rules with no
`scramble-prefix` are ordered first to match text that must remain cleartext.
See the `sr_os_rules.yml` file for the SR OS scrambling rules that are
included with the application.

Each scrambling rule can be enabled or disabled, depending on your
requirements. All rules are enabled by default, ensuring that the maximum
amount of information is scrambled. For example, the IP and ASN rules could be
disabled to allow Nokia technical support to load the configuration file on a
router when replicating a support issue, while still scrambling other
identifiable information within quoted strings.

> Caution: Disabling the `quoted-string` rule is not recommended, as it
> scrambles the largest amount of identifiable information.

### `description` (no default)

This informational option is ignored by the application. It can be used for a
free-form description or for other information about the rule. YAML `#`
characters can also be used to annotate rules.

### `enabled` (default `true`)

When `enabled: true` is set or ommitted, the rule is enabled. Rules can be
disabled by setting the value to `false`.

### `ignore-match-prefix` (no default)

When an `ignore-match-prefix` is defined and the match pattern begins with
that prefix, it is ignored before hashing so only the remaining string is
hashed. This makes the hash independent of the prefix, and the ignored prefix
is displayed at the beginning of the scrambled string output. For example,
matching `peer-as 65150` with `ignore-match-prefix: 'peer-as '` and
`scramble-prefix: 'ASN-'` displays `peer-as ASN-<hash of 65150>`. The first
time such a match occurs, it is retained as a dynamic runtime rule in memory
that matches that exact value. For example, when a string `65150` is present
in the input, it is scrambled like `ASN-<hash of 65150>` without the
`peer-as ` prefix. The dynamic rule uses the parent rule's `scramble-prefix`,
`scramble-suffix`, and `output-padding`.

### `match-pattern` (no default)

The pattern to match as defined by the `pattern-format`.

### `output-padding` (default `false`)

When `output-padding: true` is set, the output is extended so that the
scrambled string is the same length as the cleartext string. If the cleartext
string is longer than the 64-character hash, the scrambled string is padded
with lowercase alphanumeric characters (`a-z0-9`). The length of
the`scramble-prefix` and `scramble-suffix` count towards the scrambled string
length.

### `pattern-format` (default `yang`)

Two `pattern-format` types are supported:

- `yang`: the XML Schema (XSD) regex flavor used by YANG `pattern` statements,
  including multifragment concatenation with `+`, so that patterns can be
  copied from a YANG model
- `posix`: Python regular expression, without anchoring

### `scramble-group` (no default - whole match)

The `scramble-group` defines a capture group index so that only the string in
that capture group is scrambled. The rest of the string in the `match-pattern`
matched is displayed unchanged. Scramble groups are useful when the
surrounding string is needed to locate the value unambiguously but must be
preserved. For example, matching the string `filter-id="4001"` with
`match-pattern: '="([^"]*)"'`, `scramble-group: 1`, and
`scramble-prefix: 'STR-'` displays `filter-id="STR-<hash of 4001>"`, leaving
the `="` and `"` delimiters in the output.

### `scramble-prefix` (no default)

The `scramble-prefix` defines the prefix of the scrambled string, for example
`STR-`. The first 8-characters of the hash are appended to the prefix to
generate the scrambled string. When `output-padding` is enabled, the output is
padded so the scrambled string is the same length as the cleartext string.

Omit to ignore a match so that it is not scrambled and is displayed in
cleartext.

### `scramble-suffix` (no default)

The `scramble-suffix` defines the suffix of the scrambled string, for example,
characters needed to reconstruct the cleartext format.

## Scrambling examples

Example configuration files in model-driven and classic format, and other
basic examples are included in the `examples` directory. As a quick example to
get started, execute these commands to scramble an example file and display
the differences:

```sh
id_scrambler examples/md-config.cfg
diff -u0 examples/md-config.cfg examples/md-config.cfg.scrm
```

### Key requirements

A key with the `-k/--key` option may be entered with printable UTF-8
characters and must be 32 to 128 bytes long. If a key is not specified, a
random 32-byte key is generated.

> WARNING: Use a key with entropy when it is entered and not generated by the
> application.

### Scambling a file

The most common use case is when the application defaults can be used to
scramble a file. Execute this command:

```sh
id_scrambler myinputfile
```

The YAML rules in the present workding directory will be used with a generated
key, and output will be written to `./myinputfile.scrm`. Additional
application options may be specified using the command line options.

### Scrambling input interactively

Interactive scrambling is useful to quickly scramble a few lines that are
copied and pasted, or to see how an input line is scrambled. To scramble input
interactively, execute this command and then paste or enter the cleartext to
be scrambled:

```sh
id_scrambler -o stdout
```

For example:

```sh
id_scrambler -o stdout
autonomous-system 64511        // pasted input
autonomous-system ASN-304becc1 // scrambled output
stdin : Scrambled 1 string     // total for the input, reported at end of input
```

### Scrambling debug mode

Debug mode can be enabled to determine why input matches or does not match a
scrambling rule. To enable debug mode, execute this command and then paste or
enter the cleartext to be scrambled:

```sh
id_scrambler -o stdout -d
```

#### Example:

```sh
id_scrambler -o stdout -d
id_scrambler: loaded 30 rule(s) from ./sr_os_rules.yml
id_scrambler:   rule 'ignore-echo' [enabled, ignore]
id_scrambler:   rule 'ignore-command-names' [enabled, ignore]
id_scrambler:   rule 'ignore-commit-history-logs' [enabled, ignore]
id_scrambler:   rule 'ignore-numeric-entries' [enabled, ignore]
id_scrambler:   rule 'ignore-forwarding-classes' [enabled, ignore]
id_scrambler:   rule 'ignore-timos-banner' [enabled, ignore]
id_scrambler:   rule 'ignore-ipv4-well-knowns' [enabled, ignore]
id_scrambler:   rule 'ignore-ipv4-loopbacks' [enabled, ignore]
id_scrambler:   rule 'ignore-ipv6-well-knowns' [enabled, ignore]
id_scrambler:   rule 'ignore-common-keywords' [enabled, ignore]
id_scrambler:   rule 'ignore-log-messages' [disabled, ignore]
id_scrambler:   rule 'mac-address' [enabled, scramble]
id_scrambler:   rule 'ipv4-prefix' [enabled, scramble]
id_scrambler:   rule 'ipv4-address' [enabled, scramble]
id_scrambler:   rule 'ipv6-prefix' [enabled, scramble]
id_scrambler:   rule 'ipv6-address' [enabled, scramble]
id_scrambler:   rule 'autonomous-system' [enabled, scramble]
id_scrambler:   rule 'peer-as' [enabled, scramble]
id_scrambler:   rule 'local-as' [enabled, scramble]
id_scrambler:   rule 'as-colon' [enabled, scramble]
id_scrambler:   rule 'route-distinguisher' [enabled, scramble]
id_scrambler:   rule 'rd-entry' [enabled, scramble]
id_scrambler:   rule 'primary-config' [enabled, scramble]
id_scrambler:   rule 'secondary-config' [enabled, scramble]
id_scrambler:   rule 'tertiary-config' [enabled, scramble]
id_scrambler:   rule 'dns-domain' [enabled, scramble]
id_scrambler:   rule 'by-user-from' [enabled, scramble]
id_scrambler:   rule 'by-user-parentheses' [enabled, scramble]
id_scrambler:   rule 'configuration-comment' [enabled, scramble]
id_scrambler:   rule 'quoted-string' [enabled, scramble]
id_scrambler: no key supplied, generated a random 32-byte key
/configure router "Base" bgp group "IBGP"         // pasted input
CLEARTEXT : /configure router "Base" bgp group "IBGP"
RULE: ignore-common-keywords MATCHED: '"Base"' IGNORED (kept cleartext)
RULE: quoted-string MATCHED: '"IBGP"' REPLACED WITH: '"STR-2714a2f2"'
SCRAMBLED : /configure router "Base" bgp group "STR-2714a2f2"
/configure router "Base" bgp group "STR-2714a2f2" // scrambled output
stdin : Scrambled 1 string                        // total for the input
```

The input and output may also be a file.

### Scrambling using a Docker image

To run the Docker image to scramble a file, execute this command:

```sh
sudo docker run --rm -v "$PWD:/work" -u "$(id -u):$(id -g)" ghcr.io/nokia-steward/id-scrambler:1.0.0-amd64 -c /usr/share/id_scrambler/sr_os_rules.yml myinputfile
```

To run the Docker image interactively, optionally with the `-d` option to
enable debugging, execute this command:

```sh
sudo docker run --rm -i ghcr.io/nokia-steward/id-scrambler:1.0.0-amd64 -c /usr/share/id_scrambler/sr_os_rules.yml -o stdout
```

## Input file formatting

Some SR OS output uses DOS format with CRLF line endings. The application
removes all unprintable characters, such as ^M (carrige return) and ^G (bell),
from the output. This removal will cause significant differences in the output
when using tools like `diff`.

Nokia recommends converting the input file from DOS to Unix format before
scrambling. This can be done using tools like `dos2unix`, or by removing the
^M and ^G characters with tools such as `sed` or `vi`.

## Reporting issues

Nokia welcomes all issues, suggestions, and enhancements. Please submit these
through the Nokia support process. While issues raised on GitHub may be
considered for project inclusion by Nokia, direct pull requests are not
accepted.

## Known limitations

- When a scrambled string is longer than the cleartext string, output in
  `show` and `tools` commands is misaligned.
- Truncated output, for example an IP address `10.13*` will not match a
  pattern and will not be scrambled.
- Some patterns in quoted strings may match a rule before the `quoted-string`
  rule at the end, and are not scrambled. For example:
  `file-url "sftp://10.10.10.1/test"` scrambles to
  `file-url "sftp://IPv4-A-d69e5409/test"` because it matches the
  `ipv4-address` rule.
- YANG/XSD `\p{...}` translation covers `\p{L}`, `\p{N}`, `\p{Nd}`. Inside a
  character class these fall back to ASCII ranges (Python cannot nest
  classes).
- XSD character-class subtraction and the `\i`/`\c` XML name escapes are not
  supported and raise a configuration error.
- YANG patterns are used as searches rather than whole-string matches. The
  translated expression is wrapped in ASCII alphanumeric/underscore token
  boundaries, not `^` and `$` anchors.
