6.17. Tokenisers

Tokenise various languages.

Start python section to interscript/tokenisers/__init__.py[1 /1 ]

     1: #line 4 "tokenizers.ipk"
     2: # tokenisers package
     3:

End python section to interscript/tokenisers/__init__.py[1]