API

This should work .

API extracted by Sphinx

class token_utils.token_class.Token(token)

Token as generated from Python’s tokenize.generate_tokens written here in a more convenient form, and with some custom methods.

The various parameters are:

type: token type
string: the token written as a string
start = (start_row, start_col)
end = (end_row, end_col)
line: entire line of code where the token is found.

Token instances are mutable objects. Therefore, given a list of tokens, we can change the value of any token’s attribute, untokenize the list and automatically obtain a transformed source.

__contains__(str_arg)

Returns True if the string argument is a substring of the token string attribute

__eq__(other)

Compares a Token with another object; returns true if self.string == other.string or if self.string == other.

__hash__ = None

__init__(token)

Initializes using a token produced by Python’s tokenize function as input.

__len__()

Returns the length of the string attribute

__repr__()

Nicely formatted token to help with debugging session.

Note that it does not print a string representation that could be used to create a new Token instance, which is something you should never need to do other than indirectly by using the functions provided in this module.

__str__()

Returns the string attribute.

__weakref__

list of weak references to the object (if defined)

is_comment()

Returns True if the token is a comment.

is_complex()

Returns True if the token represents a complex number.cavie

is_f_string()

Return True if the token is an f-string

is_float()

Returns True if the token represents a float.

is_identifier()

Returns True if the token represents a valid Python identifier excluding Python keywords.

Note: this is different from Python’s string method isidentifier which also returns True if the string is a keyword.

is_immediately_after(other)

Returns True if the current token is immediately after other, without any intervening space in between the two tokens.

is_immediately_before(other)

Returns True if the current token is immediately before other, without any intervening space in between the two tokens.

is_in(sequence_of_strings)

Returns True if the token string is found in the sequence of strings.

is_indentation()

Returns True if the token indicates a change in indentation, (INDENT, DEDENT, BAD_DEDENT).

is_integer()

Returns True if the token represents an integer

is_keyword()

Returns True if the token represents a Python keyword.

is_name()

Returns True if the token is a type NAME

is_newline()

Returns True if the token type is either NEWLINE or NL.

is_number()

Returns True if the token represents a number.

is_operator() → bool

Returns true if the token is of type OP

is_space()

Returns True if the token indicates a change in indentation, the end of a line, or the end of the source (INDENT, DEDENT, BAD_DEDENT, NEWLINE, NL, and ENDMARKER).

Note that spaces, including tab characters \t, between tokens on a given line are not considered to be tokens themselves.

is_string()

Returns True if the token represents a string

is_unclosed_string()

Returns True if the token is an unclosed string

token_utils.token_class.make_fake_token(type=-4, string='$', start=(0, 0), end=(0, 0), line='')

Useful when we need to process a list of tokens with multiple consecutive at a time, and we need to lengthen the list for doing so.

Do not use as token to be inserted in a list of tokens to be untokenize as it will almost certainly not lead to the desired result. If needed for modifying a list of token prior to untokenizing, simply insert regular strings instead of fake tokens.