About tokens and token-utils

About tokens

Using token-utils requires knowing what “tokens” produced by Python’s tokenize module are. An outstanding tutorial about this module is Brown Water Python written by Aaron Meurer.

Origin of token-utils

In our description of these tools below, we assume that you are somewhat familiar with the concept of token objects generated by Python’s tokenize module. If you are not familiar with those, we suggest that you read through at least once through the documentation about Python’s tokenize module mentioned above.

An excellent tutorial

Brown Water Python, written by Aaron Meurer,

is an excellent and very comprehensive tutorial about Python’s

tokenize module.

The main points to understand:

• Using the tokenize function, a source can be broken down in tokens, which, as generated by Python, are 5-tuples carrying information about their type, their string content, their position in the source (identified by starting and ending row, aka line number, and column), as well as the content of the line where they are found.

• From a list of tokens, the original source can essentially recreated by using the untokenize function. However, as stated in the documentation:

  The result is guaranteed to tokenize back to match the input so that the conversion is lossless and round-trips are assured. The guarantee applies only to the token type and token string as the spacing between tokens (column positions) may change.

• To untokenize using the function from the Python standard library, one can use either a list of 5-tuple tokens, or a list of two-tuple tokens that include only the type and string information.

Perfect round-trip

Unlike Python’s version, the process of tokenizing and untokenizing a source using ideas’ own tokenize and untokenize functions is guaranteed to yield back an exact copy of the original source, with all the spacing information intact. Experience has shown that being able to recover the original source with spacing included is extremely useful when writing tests about the expected results for some source transformation.

Tip

While we show below the full API of the token_utils module, you might want to first to to next page to see a demonstration of its usage, done in an actual programming session using a Jupyter notebook.