Origin and history of token-utils

This is likely going to be of very little interest to anyone, except perhaps for budding programmers who are not sure that the programming experiments they do are worthwhile. If this is you, I have a message: yes, anything you do for learning is worthwhile.

Note to self?

As I was writing this documentation, I often felt the need to explain why a certain feature of token-utils existed, going into some details about the reasoning for it. This lead me, mostly for my own curiosity, to try to figure out how it began and evolve. As I was starting to put things together, I thought I should write down the full history.

Error

The rest of this document is currently just a “brain dump” with various links.

A coherent text will eventually be written.

Talk about starting to program in 2004, wanting to motivate my children to learn programming using a desktop application I created to teach myself Python: RUR-PLE RUR-PLE eventually supported 8 languages. It has been used (and still is!) in many schools, including many elementary schools in Korea with books written by Samsung as I reported on my blog in 2014. In that same blog post, I announced the creation of Reeborg’s World a significantly improved version of RUR-PLE, running entirely in a browser, and thus removing the need to install anything on a computer. Reeborg’s World is still very much in use today. Two months later, I announced the availability of a French version, where the basic robot commands, such as move(), turn_left(), etc., were translated, but the rest was still straight Python.

Somewhere along the way, I added the possibility to write:

repeat n:
   # code block

as equivalent to:

for _ in range(n):
   # code block

In 2015, I started to play with import hooks how to add similar new constructs in Python.

https://aroberge.blogspot.com/2015/10/from-experimental-import-somethingnew.html

In 2018, I started working on AvantPy AvantPy is now defunct as I found out that Hedy did such a much better job at teaching programming using Python-like constructs in a huge number of human languages.

I wrote token-utils first as a module integrated within ideas

Before I completely shelved AvantPy, I started working on friendly/friendly-traceback in 2019.

In 2020, I started working on ideas. I wrote at the time in the readme file

As I find myself doing a lot of copy/paste/modify on the various import hooks experiments, including on some published projects such as AvantPy, I thought it would make sense to create a versatile projects which I could use as the basis of other projects. An obvious benefit is that I will need to fix bugs in a single project.

I was partly inspired by the following:

In Feb. 21, 2020, on the Python-ideas mailing list, Andrew Barnert, a Python code developer, wrote:

Unfortunately, the boilerplate to write an import hook is more complicated than you’d like (and pretty hard to figure out the first time), and the support for filtering on the token stream (the most obvious way to do this one) rather than the text stream, AST, or bytecode is pretty minimal and clumsy. [emphasis added]

I made some quick progress, and initially created token-utils as a module within it. When I talked about ideas in a tread on Python-ideas, the same Andrew Barnert

also wrote to a message writing the following about token-utils as an existing module within ideas :

First, one of the parts that always blocked me was coming up with a good way to write token-level transformers. It looks like you solved that; maybe it’s worth pulling the tokenizer tools out into a separate library to post on PyPI separately?

This motivated me to turn token-utils on its own project available on Pypi. It quickly become mature and did not change between the end of 2020 and the middle of 2026.

I wrote token-utils first as a module integrated within ideas

While ideas aims to provide support for all kinds of transformations, including those that affect the Abstract Syntax Tree or the bytecode, most transformations deal with exploring alternative syntax that is not compatible with Python’s current syntax defined by its grammar. Such alternative syntax cannot be parsed by Python without generating a SyntaxError thus preventing the execution of the code. For this reason, almost all of our examples transform the code prior to letting Python parse it. We do this using a set of tools built upon Python’s tokenize module.

In the meantime, I incorporated a version of it (rather than having it as an external requirement) within friendly/friendly-traceback where I focused on adding methods and functions not focused on code-transformation, but rather in helping replace the dreaded SyntaxError: invalid syntax by more appropriate and helpful description of the exact cause of the problem and offering possible solutions.