token_utils.py_tokenize
Tokenization help for Python programs.
This has been adapted from Python 3.11 tokenize.py It has been mostly copied almost exactly except that anything about “untokenizing” has been removed as we rely on our own version. We’ve also removed anything related to encoding as it is not needed.
Anything in this module should NOT be called directly from your code.
Functions
|
|
|
Tokenize a source reading Python code as unicode strings. |
|
|
|
Classes
|