Re: ka-ping yee tokenizer.py

[Fredrik Lundh]

Karl Kobata wrote:

I have enjoyed using ka-ping yee’s tokenizer.py. I would like to
replace the readline parameter input with my own and pass a list of
strings to the tokenizer. I understand it must be a callable object and
iteratable but it is obvious with errors I am getting, that this is not
the only functions required.

not sure I can decipher your detailed requirements, but to use Python's
standard "tokenize" module (written by ping) on a list, you can simple
do as follows:

import tokenize

program = [ ... program given as list ... ]

for token in tokenize.genera te_tokens(iter( program).next):
print token

another approach is to turn the list back into a string, and wrap that
in a StringIO object:

import tokenize
import StringIO

program = [ ... program given as list ... ]

program_buffer = StringIO.String IO("".join(prog ram))

for token in tokenize.genera te_tokens(progr am_buffer.readl ine):
print token

</F>