The python tangler currently tokenises the generated code file (as it is being generated). The tokeniser used is the standard one, tokeniser.py. Unfortunately, it has the wrong control interface, and we have to map it to one the tangler can drive. This requires coroutines, but Python doesn't have them, so we are forced to emulate them using threads.
The python tokeniser is now fully functional.
The control system has been almost completely redone. It still uses the lists of regular expressions paired with functions to execute, but the functions are now methods bound to frames, usually the current input frame.
The predefined methods are replaced for every input frame constructed. This means users cannot override them properly, since they're not inherited. Really, we should use the Guido hack: store functions in the user dictionary, and turn them into bound methods at the point of lookup.
Correct implementation of the scoping mechanism is severely hampered by Python's lack of suitable scoping.