Metadata-Version: 2.4
Name: token-utils
Version: 0.2.1
Summary: Convenient tools to deal with tokens.
Project-URL: Documentation, https://aroberge.github.io/token-utils/docs/html/
Project-URL: Issues, https://github.com/aroberge/token-utils/issues
Project-URL: Source, https://github.com/aroberge/token-utils
Author-email: André Roberge <andre.roberge@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Python: >=3.6
Description-Content-Type: text/markdown

# token-utils

Installation: `pip install token-utils`.

The purpose of token-utils is to simplify manipulations of tokens normally
obtaind from Python's [tokenize module](https://docs.python.org/3/library/tokenize.html).
One of token-utils' features is that, unlike Python's version, the following
is always guaranteed:

```python
from token_utils import tokenize, untokenize

source = "Arbitrary Python code here"

assert source == untokenize(tokenize(source))
```

To get an idea of the simplicity of using token-utils,
consider [this example from Python's standard library](https://docs.python.org/3/library/tokenize.html#examples) which substitute Decimals for floats in a string of statements:

```python
from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
from io import BytesIO

def decistmt(s):
    result = []
    g = tokenize(BytesIO(s.encode('utf-8')).readline)  # tokenize the string
    for toknum, tokval, _, _, _ in g:
        if toknum == NUMBER and '.' in tokval:  # replace NUMBER tokens
            result.extend([
                (NAME, 'Decimal'),
                (OP, '('),
                (STRING, repr(tokval)),
                (OP, ')')
            ])
        else:
            result.append((toknum, tokval))
    return untokenize(result).decode('utf-8')
```

Here's how you could achieve the same result with token-utils:

```python
from token_utils import tokenize, untokenize

def decistmt(source):
    tokens = tokenize(source)
    for token in tokens:
        if token.is_float():
            token.string = f"Decimal('{token.string}')"
    return untokenize(tokens)
```

See [the documentation](https://aroberge.github.io/token-utils/docs/html/) for more information.
