API

token_utils.py API extracted by Sphinx

class token_utils.Token(token)

Token as generated from Python’s tokenize.generate_tokens written here in a more convenient form, and with some custom methods.

The various parameters are:

type: token type
string: the token written as a string
start = (start_row, start_col)
end = (end_row, end_col)
line: entire line of code where the token is found.

Token instances are mutable objects. Therefore, given a list of tokens, we can change the value of any token’s attribute, untokenize the list and automatically obtain a transformed source.

__contains__(str_arg)

Returns True if the string argument is a substring of the token string attribute

__eq__(other)

Compares a Token with another object; returns true if self.string == other.string or if self.string == other.

__hash__ = None

__init__(token)

__repr__()

Nicely formatted token to help with debugging session.

Note that it does not print a string representation that could be used to create a new Token instance, which is something you should never need to do other than indirectly by using the functions provided in this module.

__str__()

Returns the string attribute.

__weakref__

list of weak references to the object (if defined)

is_comment()

Returns True if the token is a comment.

is_complex()

Returns True if the token represents a complex number

is_float()

Returns True if the token represents a float

is_identifier()

Returns True if the token represents a valid Python identifier excluding Python keywords.

Note: this is different from Python’s string method isidentifier which also returns True if the string is a keyword.

is_integer()

Returns True if the token represents an integer

is_keyword()

Returns True if the token represents a Python keyword.

is_name()

Returns True if the token is a type NAME

is_number()

Returns True if the token represents a number

is_space()

Returns True if the token indicates a change in indentation, the end of a line, or the end of the source (INDENT, DEDENT, NEWLINE, NL, and ENDMARKER).

Note that spaces, including tab characters \t, between tokens on a given line are not considered to be tokens themselves.

is_string()

Returns True if the token is a string

token_utils.dedent(tokens, nb)

Given a list of tokens, produces an equivalent list corresponding to a line of code with the first nb characters removed.

token_utils.find_substring_index(main, substring)

Somewhat similar to the find() method for strings, this function determines if the tokens for substring appear as a subsequence of the tokens for main. If so, the index of the first token in returned, otherwise -1 is returned.

token_utils.find_token_by_position(tokens, row, column)

Given a list of tokens, a specific row (linenumber) and column, a two-tuple is returned that includes the token found at that position as well as its list index.

If no such token can be found, None, None is returned.

token_utils.fix_empty_line(source, tokens)

Prior to version 3.12, Python’s tokenizer drops entirely a last line if it consists only of space characters and/or tab characters.

To ensure that we can always have:

untokenize(tokenize(source)) == source

we correct the last token content if needed.

token_utils.get_first(tokens, exclude_comment=True)

Given a list of tokens, find the first token which is not a space token (such as a NEWLINE, INDENT, DEDENT, etc.) and, by default, also not a COMMMENT.

COMMMENT tokens can be included by setting exclude_comment to False.

Returns None if none is found.

token_utils.get_first_index(tokens, exclude_comment=True)

Given a list of tokens, find the index of the first token which is not a space token (such as a NEWLINE, INDENT, DEDENT, etc.) nor a COMMMENT. If it is desired to include COMMENT, set exclude_comment to True.

Returns None if none is found.

token_utils.get_last(tokens, exclude_comment=True)

Given a list of tokens, find the last token which is not a space token (such as a NEWLINE, INDENT, DEDENT, etc.) and, by default, also not a COMMMENT.

COMMMENT tokens can be included by setting``exclude_comment`` to False.

Returns None if none is found.

token_utils.get_last_index(tokens, exclude_comment=True)

Given a list of tokens, find the index of the last token which is not a space token (such as a NEWLINE, INDENT, DEDENT, etc.) nor a COMMMENT. If it is desired to include COMMENT, set exclude_comment to True.

Returns None if none is found.

token_utils.get_lines(source)

Transforms a source (string) into a list of Tokens, with each (inner) list containing all the tokens found on a given line of code.

token_utils.get_number(tokens, exclude_comment=True)

Given a list of tokens, gives a count of the number of tokens which are not space tokens (such as NEWLINE, INDENT, DEDENT, etc.)

By default, COMMMENT tokens are not included in the count. If you wish to include them, set exclude_comment to False.

token_utils.get_significant_tokens(source)

Gets a list of tokens from a source (str), ignoring comments as well as any token whose string value is either null or consists of spaces, newline or tab characters.

If an exception is raised by Python’s tokenize module, the list of tokens accumulated up to that point is returned.

token_utils.indent(tokens, nb, tab=False)

Given a list of tokens, produces an equivalent list corresponding to a line of code with nb space characters inserted at the beginning.

If tab is specified to be True, nb tab characters are inserted instead of spaces.

token_utils.print_tokens(source)

Prints tokens found in source, excluding spaces and comments.

source is either a string to be tokenized, or a list of Token objects.

This is occasionally useful as a debugging tool.

token_utils.strip_comment(line)

Removes comments from a line

token_utils.tokenize(source, warning=True)

Transforms a source (string) into a list of Tokens.

If an exception is raised by Python’s tokenize module, the list of tokens accumulated up to that point is returned.

token_utils.untokenize(tokens)

Return source code based on tokens.

Adapted from https://github.com/myint/untokenize, Copyright (C) 2013-2018 Steven Myint, MIT License (same as this project).

This is similar to Python’s own tokenize.untokenize(), except that it preserves spacing between tokens, by using the line information recorded by Python’s tokenize.generate_tokens. As a result, if the original soure code had multiple spaces between some tokens or if escaped newlines were used or if tab characters were present in the original source, those will also be present in the source code produced by untokenize.

Thus source == untokenize(tokenize(source)).

Note: if you you modifying tokens from an original source:

Instead of full token object, untokenize will accept simple strings; however, it will only insert them as is without taking them into account when it comes with figuring out spacing between tokens.