Removing comments... tokenize error

[qwweeeit]

In analysing a very big application (pysol) made of almost
100 sources, I had the need to remove comments.

Removing the comments which take all the line is straightforward ...

Instead for the embedded comments I used the tokenize module.

To my surprise the analysed output is different from the input
(the last tuple element should exactly replicate the input line)
The error comes out in correspondance of a triple string.
I don't know if this has already been corrected (I use Python 2.3)
or perhaps is a mistake on my part...

Next you find the script I use to replicate the strange behaviour:

import tokenize

Input = "pippo1"
Output = "pippo2"

f = open(Input)
fOut=open(Outpu t,"w")

nLastLine=0
for i in tokenize.genera te_tokens(f.rea dline):
.. if nLastLine != (i[2])[0]: # the 3rd element of the tuple is
.. . nLastLine = (i[2])[0] # (startingRow, startingCol)
.. . fOut.write(i[4])

f.close()
fOut.close()

The file to be used (pippo1) contains an extract:

class SelectDialogTre eData:
.. img = None
.. def __init__(self):
.. . self.tree_xview = (0.0, 1.0)
.. . self.tree_yview = (0.0, 1.0)
.. . if self.img is None:
.. . . SelectDialogTre eData.img = (makeImage(dith er=0, data="""
R0lGODlhEAAOAPI FAAAAAICAgMDAwP//AP///4AAAAAAAAAAACH5 BAEAAAUALAAAAAA QAA4AAAOL
WLrcGxA6FoYYYoR ZwhCDMAhDFCkBoa 6sGgBFQAzCIAzCI AzCEACFAEEwEAwE A8FAMBAEAIUAYSA Y
CAaCgWAgGAQAhQB BMBAMBAPBQDAQBA CFAGEgGAgGgoFgI BgEAAUBBAIDAgMC AwIDAgMCAQAFAQQ D
AgMCAwIDAgMCAwE ABSaiogAKAKeoqa kFCQA7"""), makeImage(dithe r=0, data="""
R0lGODlhEAAOAPI FAAAAAICAgMDAwP//AP///4AAAAAAAAAAACH5 BAEAAAUALAAAAAA QAA4AAAN3
WLrcHBA6Foi1YZZ AxBCDQESREhCDMA iDcFkBUASEMAiDM AiDMAgBAGlIGgQA gZeSEAAIAoAAQTA Q
DAQDwUAwAEAAhQB BMBAMBAPBQBAABA CFAGEgGAgGgoFgI AAEAAoBBAMCAwID AgMCAwEAAApERI4 L
jpWWlgkAOw==""" ), makeImage(dithe r=0, data="""
R0lGODdhEAAOAPI AAAAAAAAAgICAgM DAwP///wAAAAAAAAAAACwA AAAAEAAOAAADTii 63DowyiiA
GCHrnQUQAxcQAAE QgAAIg+MCwkDMdD 0LgDDUQG8LAMGg1 gPYBADBgFbs1QQA wYDWBNQEAMHABrA R
BADBwOsVAFzoqlq dAAA7"""), makeImage(dithe r=0, data="""
R0lGODdhEAAOAPI AAAAAAAAAgICAgM DAwP8AAP///wAAAAAAACwAAAAA EAAOAAADVCi63Do wyiiA
GCHrnQUQAxcUQAE UgAAIg+MCwlDMdD 0LgDDQBE3UAoBgU CMUCDYBQDCwEWwF AUAwqBEKBJsAIBj Q
CDRCTQAQDKBQAcD FBrjf8Lg7AQA7"" "))

The output of tokenize (pippo2) gives instead:

class SelectDialogTre eData:
.. img = None
.. def __init__(self):
.. . self.tree_xview = (0.0, 1.0)
.. . self.tree_yview = (0.0, 1.0)
.. . if self.img is None:
.. . . SelectDialogTre eData.img = (makeImage(dith er=0, data="""
AgMCAwIDAgMCAwE ABSaiogAKAKeoqa kFCQA7"""), makeImage(dithe r=0, data="""
jpWWlgkAOw==""" ), makeImage(dithe r=0, data="""
BADBwOsVAFzoqlq dAAA7"""), makeImage(dithe r=0, data="""
CDRCTQAQDKBQAcD FBrjf8Lg7AQA7"" "))

.... with a big difference! Why?

[Fredrik Lundh]

Re: Removing comments... tokenize error

"qwweeeit" <qwweeeit@yahoo .it> wrote:
[color=blue]
> I don't know if this has already been corrected (I use Python 2.3)
> or perhaps is a mistake on my part...[/color]

it's a mistake on your part. adding a print statement to the for-
loop might help you figure it out:
[color=blue]
> nLastLine=0
> for i in tokenize.genera te_tokens(f.rea dline):[/color]
print i[color=blue]
> . if nLastLine != (i[2])[0]: # the 3rd element of the tuple is
> . . nLastLine = (i[2])[0] # (startingRow, startingCol)
> . . fOut.write(i[4])[/color]

(hints: what happens if a token spans multiple lines? and how does
the tokenize module deal with comments?)

</F>

[qwweeeit]

Re: Removing comments... tokenize error

Thanks! If you answer to my posts one more time I could consider you as
my tutor...

It was strange to have found a bug...! In any case I will not go deeper
into the matter, because for me it's enough your explanatiom.
I corrected the problem by hand removing the tokens spanning multiple lines
(there were only 8 cases...).

Instead I haven't understood your hint about comments...
I succeded in realizing a python script which removes comments.

Here it is (in all its cumbersome and criptic appearence!...) :

# removeCommentsT ok.py
import tokenize
Input = "pippo1"
Output = "pippo2"
f = open(Input)
fOut=open(Outpu t,"w")

nLastLine=0
for i in tokenize.genera te_tokens(f.rea dline):
.. if i[0]==52 and nLastLine != (i[2])[0]:
.. . fOut.write((i[4].replace(i[1],'')).rstrip()+ '\n')
.. . nLastLine=(i[2])[0]
.. elif i[0]==4 and nLastLine != (i[2])[0]:
.. . fOut.write((i[4]))
.. . nLastLine=(i[2])[0]
f.close()
fOut.close()

Some explanations for the guys like me...:
- 52 and 4 are the arbitrary codes for comments and NEWLINE respectively
- the comment removing is obtained by clearing the comment (i[1]) in the
input line (i[4])
- I also right trimmed the line to get rid off the remaining blanks.

[M.E.Farmer]

whitespace , comment stripper, and EOL converter

qwweeeit wrote:[color=blue]
> Thanks! If you answer to my posts one more time I could consider you[/color]
as[color=blue]
> my tutor...
>
> It was strange to have found a bug...! In any case I will not go[/color]
deeper[color=blue]
> into the matter, because for me it's enough your explanatiom.
> I corrected the problem by hand removing the tokens spanning multiple[/color]
lines[color=blue]
> (there were only 8 cases...).
>
> Instead I haven't understood your hint about comments...
> I succeded in realizing a python script which removes comments.
>
> Here it is (in all its cumbersome and criptic appearence!...) :
>
> # removeCommentsT ok.py
> import tokenize
> Input = "pippo1"
> Output = "pippo2"
> f = open(Input)
> fOut=open(Outpu t,"w")
>
> nLastLine=0
> for i in tokenize.genera te_tokens(f.rea dline):
> . if i[0]==52 and nLastLine != (i[2])[0]:
> . . fOut.write((i[4].replace(i[1],'')).rstrip()+ '\n')
> . . nLastLine=(i[2])[0]
> . elif i[0]==4 and nLastLine != (i[2])[0]:
> . . fOut.write((i[4]))
> . . nLastLine=(i[2])[0]
> f.close()
> fOut.close()
>
> Some explanations for the guys like me...:
> - 52 and 4 are the arbitrary codes for comments and NEWLINE[/color]
respectively[color=blue]
> - the comment removing is obtained by clearing the comment (i[1]) in[/color]
the[color=blue]
> input line (i[4])
> - I also right trimmed the line to get rid off the remaining blanks.[/color]
Tokenizer sends multiline strings and comments as a single token.

############### ############### ############### ############### ##########
# python comment and whitespace stripper :)
############### ############### ############### ############### ##########

import keyword, os, sys, traceback
import StringIO
import token, tokenize
__credits__ = 'just another tool that I needed'
__version__ = '.7'
__author__ = 'M.E.Farmer'
__date__ = 'Jan 15 2005, Oct 24 2004'

############### ############### ############### ############### ##########

class Stripper:
"""python comment and whitespace stripper :)
"""
def __init__(self, raw):
self.raw = raw

def format(self, out=sys.stdout, comments=0, spaces=1,
untabify=1, eol='unix'):
''' strip comments, strip extra whitespace,
convert EOL's from Python code.
'''
# Store line offsets in self.lines
self.lines = [0, 0]
pos = 0
# Strips the first blank line if 1
self.lasttoken = 1
self.temp = StringIO.String IO()
self.spaces = spaces
self.comments = comments

if untabify:
self.raw = self.raw.expand tabs()
self.raw = self.raw.rstrip ()+' '
self.out = out

self.raw = self.raw.replac e('\r\n', '\n')
self.raw = self.raw.replac e('\r', '\n')
self.lineend = '\n'

# Gather lines
while 1:
pos = self.raw.find(s elf.lineend, pos) + 1
if not pos: break
self.lines.appe nd(pos)

self.lines.appe nd(len(self.raw ))
# Wrap text in a filelike object
self.pos = 0

text = StringIO.String IO(self.raw)

# Parse the source.
## Tokenize calls the __call__
## function for each token till done.
try:
tokenize.tokeni ze(text.readlin e, self)
except tokenize.TokenE rror, ex:
traceback.print _exc()

# Ok now we write it to a file
# but we also need to clean the whitespace
# between the lines and at the ends.
self.temp.seek( 0)

# Mac CR
if eol == 'mac':
self.lineend = '\r'
# Windows CR LF
elif eol == 'win':
self.lineend = '\r\n'
# Unix LF
else:
self.lineend = '\n'

for line in self.temp.readl ines():
if spaces == -1:
self.out.write( line.rstrip()+s elf.lineend)
else:
if not line.isspace():
self.lasttoken= 0
self.out.write( line.rstrip()+s elf.lineend)
else:
self.lasttoken+ =1
if self.lasttoken< =self.spaces and self.spaces:
self.out.write( self.lineend)


def __call__(self, toktype, toktext,
(srow,scol), (erow,ecol), line):
''' Token handler.
'''
# calculate new positions
oldpos = self.pos
newpos = self.lines[srow] + scol
self.pos = newpos + len(toktext)

#kill the comments
if not self.comments:
# Kill the comments ?
if toktype == tokenize.COMMEN T:
return

# handle newlines
if toktype in [token.NEWLINE, tokenize.NL]:
self.temp.write (self.lineend)
return

# send the original whitespace, if needed
if newpos > oldpos:
self.temp.write (self.raw[oldpos:newpos])

# skip indenting tokens
if toktype in [token.INDENT, token.DEDENT]:
self.pos = newpos
return

# send text to the temp file
self.temp.write (toktext)
return
############### ############### ############### ############### ##########

def Main():
import sys
if sys.argv[1]:
filein = open(sys.argv[1]).read()
Stripper(filein ).format(out=sy s.stdout, comments=1, untabify=1,
eol='win')

############### ############### ############### ############### ##########

if __name__ == '__main__':
Main()

M.E.Farmer

[qwweeeit]

Re: Removing comments... tokenize error

My code, besides beeing cumbersome and criptic, has another quality:
it is
buggy!
I apologize for that; obviously I discovered it after posting (in the
best tradition of Murphy's law!).
When I will find the solution I let you know, also if the problem is
made difficult for the fact that the for cycle is indexed in terms of
a 5 element tuple, not very easy (at least for me!...).

[qwweeeit]

Re: whitespace , comment stripper, and EOL converter

Hi,
I have no more need to corret my code's bugs and send to clp group a
working application (I don't think that there was an eager
expectation...) .
Your code is perfectly working (as you can expect from a guru...).
Thank you and bye.

[qwweeeit]

Re: whitespace , comment stripper, and EOL converter

Hi,

At last I succeded in implementing a cross reference tool!
(with your help and that of other gurus...).
Now I can face the problem (for me...) of understanding your
code (I have not grasped the classes and objects...).

I give you a brief example of the xref output (taken from your code,
also if the line numbers don't match, because I modified your code,
not beeing interested in eof's other than Linux).

and 076 if self.lasttoken< =self.spaces and
self.spaces:
append 046 self.lines.appe nd(pos)
append 048 self.lines.appe nd(len(self.raw ))
argv 116 if sys.argv[1]:
argv 117 filein = open(sys.argv[1]).read()
__author__ 010 __author__ = s_
break 045 if not pos: break
__call__ 080 def __call__(self, toktype, toktext, (srow,scol),
.. .
(erow,ecol), line):
class 015 class Stripper:
COMMENT 092 if toktype == tokenize.COMMEN T:
comments 021 def format(self, out=sys.stdout, comments=0,
spaces=1,untabi fy=1):
comments 033 self.comments = comments
comments 090 if not self.comments:
comments 118 Stripper(filein ).format(out=sy s.stdout,
comments=0, .
untabify=1)
__credits__ 008 __credits__ = s_
__date__ 011 __date__ = s_
DEDENT 105 if toktype in [token.INDENT, token.DEDENT]:
def 018 def __init__(self, raw):
def 021 def format(self, out=sys.stdout, comments=0,
.. spaces=1,untabi fy=1):
def 080 def __call__(self, toktype, toktext, (srow,scol),
(erow,ecol), line):
def 114 def Main():
ecol 080 def __call__(self, toktype, toktext, (srow,scol),
.. (erow,ecol), line):
erow 080 def __call__(self, toktype, toktext, (srow,scol),
.. (erow,ecol), line):
ex 059 except tokenize.TokenE rror, ex:
except 059 except tokenize.TokenE rror, ex:
expandtabs 036 self.raw = self.raw.expand tabs()
filein 117 filein = open(sys.argv[1]).read()
filein 118 Stripper(filein ).format(out=sy s.stdout,
comments=0,

untabify=1)
find 044 pos = self.raw.find(s elf.lineend, pos) + 1
format 021 def format(self, out=sys.stdout, comments=0,
spaces=1,untabi fy=1):
format 118 Stripper(filein ).format(out=sy s.stdout,
comments=0,
untabify=1)
import 005 import keyword, os, sys, traceback
import 006 import StringIO
import 007 import token, tokenize
import 115 import sys
INDENT 105 if toktype in [token.INDENT, token.DEDENT]:
__init__ 018 def __init__(self, raw):
isspace 071 if not line.isspace():
keyword 005 import keyword, os, sys, traceback
lasttoken 030 self.lasttoken = 1
lasttoken 072 self.lasttoken= 0
lasttoken 075 self.lasttoken+ =1
lasttoken 076 if self.lasttoken< =self.spaces and
self.spaces:
....

To obtain this output, you must remove comments and empty lines, move
strings in a db file, leaving as place holder s_ for normal strings
and m_ for triple strings.
See an example:

m_ """python comment and whitespace stripper :)""" #016
m_ ''' strip comments, strip extra whitespace, convert EOL's from
Python
code.'''#023
m_ ''' Token handler.''' #082

s_ 'just another tool that I needed' |008 __credits__ = 'just another
tool
that I needed'
s_ '.7' |009 __version__ = '.7'
s_ 'M.E.Farmer' |010 __author__ = 'M.E.Farmer'
s_ 'Jan 15 2005, Oct 24 2004' |011 __date__ = 'Jan 15 2005, Oct 24
2004'
s_ ' ' |037 self.raw = self.raw.rstrip ()+'
'
s_ '\n' |040 self.lineend = '\n'
s_ '__main__' |122 if __name__ == '__main__':

I think that this tool is very useful.

Bye

[M.E.Farmer]

Re: whitespace , comment stripper, and EOL converter

Glad you are making progress ;)
[color=blue]
>I give you a brief example of the xref output (taken from your >code,
>also if the line numbers don't match, because I modified >your code,
>not beeing interested in eof's other than Linux).[/color]

What happens when you try to analyze a script from a diffrent os ? It
usually looks like a skewed mess, that is why I have added EOL
conversion so it is painless for you to convert to your eol of choice.
The code I posted consist of a class and a Main function.
The class has three methods.
__init__ is called by Python when you create an instance of the class
Stripper. All __init__ does here is just set a class variable self.raw
..
format is called explicitly with a few arguments to start the
tokenizer.
__call__ is special it is not easy to grasp how this even works.. at
first.
In Python when you treat an instance like a function, Python invokes
the __call__method of that instance if present and if it is callable().
example:
try:
tokenize.tokeni ze(text.readlin e, self)
except tokenize.TokenE rror, ex:
traceback.print _exc()
The snippet above is from the Stripper class.
Notice that tokenize.tokeni ze is being feed a reference to self ( if
this code is running self is an instance of Stripper ).
tokenize.tokeni ze is really a hidden loop.
Each token generated is sent to self as five parts toktype, toktext,
(startrow,start col), (endrow,endcol) , and line. Self is callable and
has a __call__ method so tokenize sends really sends the five part
info to __call__ for every token.
If this was obvious then ignore it ;)

M.E.Farmer

[MrJean1]

Re: whitespace , comment stripper, and EOL converter


Great tool, indeed! But doc strings stay in the source text.

If you do need to remove doc strings as well, add the following into
the __call__ method.

.... # kill doc strings
.... if not self.docstrings :
.... if toktype == tokenize.STRING and len(toktext) >= 6:
.... t = toktext.lstrip( 'rRuU')
.... if ((t.startswith( "'''") and t.endswith("''' ")) or
.... (t.startswith(' """') and t.endswith('""" '))):
.... return

as shown in the original post below. Also, set self.docstrings in the
format method, similar to self.comments as shown below in lines
starting with '...'.


/Jean Brouwers



M.E.Farmer wrote:[color=blue]
> qwweeeit wrote:[color=green]
> > Thanks! If you answer to my posts one more time I could consider[/color][/color]
you[color=blue]
> as[color=green]
> > my tutor...
> >
> > It was strange to have found a bug...! In any case I will not go[/color]
> deeper[color=green]
> > into the matter, because for me it's enough your explanatiom.
> > I corrected the problem by hand removing the tokens spanning[/color][/color]
multiple[color=blue]
> lines[color=green]
> > (there were only 8 cases...).
> >
> > Instead I haven't understood your hint about comments...
> > I succeded in realizing a python script which removes comments.
> >
> > Here it. is (in all its cumbersome and criptic appearence!...) :
> >
> > # removeCommentsT ok.py
> > import tokenize
> > Input = "pippo1"
> > Output = "pippo2"
> > f = open(Input)
> > fOut=open(Outpu t,"w")
> >
> > nLastLine=0
> > for i in tokenize.genera te_tokens(f.rea dline):
> > . if i[0]==52 and nLastLine != (i[2])[0]:
> > . . fOut.write((i[4].replace(i[1],'')).rstrip()+ '\n')
> > . . nLastLine=(i[2])[0]
> > . elif i[0]==4 and nLastLine != (i[2])[0]:
> > . . fOut.write((i[4]))
> > . . nLastLine=(i[2])[0]
> > f.close()
> > fOut.close()
> >
> > Some explanations for the guys like me...:
> > - 52 and 4 are the arbitrary codes for comments and NEWLINE[/color]
> respectively[color=green]
> > - the comment removing is obtained by clearing the comment (i[1])[/color][/color]
in[color=blue]
> the[color=green]
> > input line (i[4])
> > - I also right trimmed the line to get rid off the remaining[/color][/color]
blanks.[color=blue]
> Tokenizer sends multiline strings and comments as a single token.
>
>[/color]
############### ############### ############### ############### ##########[color=blue]
> # python comment and whitespace stripper :)
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> import keyword, os, sys, traceback
> import StringIO
> import token, tokenize
> __credits__ = 'just another tool that I needed'
> __version__ = '.7'
> __author__ = 'M.E.Farmer'
> __date__ = 'Jan 15 2005, Oct 24 2004'
>
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> class Stripper:
> """python comment and whitespace stripper :)
> """
> def __init__(self, raw):
> self.raw = raw
>[/color]
.... def format(self, out=sys.stdout, comments=0, docstrings=0,
spaces=1,[color=blue]
> untabify=1, eol='unix'):
> ''' strip comments, strip extra whitespace,
> convert EOL's from Python code.
> '''
> # Store line offsets in self.lines
> self.lines = [0, 0]
> pos = 0
> # Strips the first blank line if 1
> self.lasttoken = 1
> self.temp = StringIO.String IO()
> self.spaces = spaces
> self.comments = comments[/color]
.... self.docstrings = docstrings[color=blue]
>
> if untabify:
> self.raw = self.raw.expand tabs()
> self.raw = self.raw.rstrip ()+' '
> self.out = out
>
> self.raw = self.raw.replac e('\r\n', '\n')
> self.raw = self.raw.replac e('\r', '\n')
> self.lineend = '\n'
>
> # Gather lines
> while 1:
> pos = self.raw.find(s elf.lineend, pos) + 1
> if not pos: break
> self.lines.appe nd(pos)
>
> self.lines.appe nd(len(self.raw ))
> # Wrap text in a filelike object
> self.pos = 0
>
> text = StringIO.String IO(self.raw)
>
> # Parse the source.
> ## Tokenize calls the __call__
> ## function for each token till done.
> try:
> tokenize.tokeni ze(text.readlin e, self)
> except tokenize.TokenE rror, ex:
> traceback.print _exc()
>
> # Ok now we write it to a file
> # but we also need to clean the whitespace
> # between the lines and at the ends.
> self.temp.seek( 0)
>
> # Mac CR
> if eol == 'mac':
> self.lineend = '\r'
> # Windows CR LF
> elif eol == 'win':
> self.lineend = '\r\n'
> # Unix LF
> else:
> self.lineend = '\n'
>
> for line in self.temp.readl ines():
> if spaces == -1:
> self.out.write( line.rstrip()+s elf.lineend)
> else:
> if not line.isspace():
> self.lasttoken= 0
> self.out.write( line.rstrip()+s elf.lineend)
> else:
> self.lasttoken+ =1
> if self.lasttoken< =self.spaces and self.spaces:
> self.out.write( self.lineend)
>
>
> def __call__(self, toktype, toktext,
> (srow,scol), (erow,ecol), line):
> ''' Token handler.
> '''
> # calculate new positions
> oldpos = self.pos
> newpos = self.lines[srow] + scol
> self.pos = newpos + len(toktext)
>
> #kill the comments
> if not self.comments:
> # Kill the comments ?
> if toktype == tokenize.COMMEN T:
> return
>[/color]
.... # kill doc strings
.... if not self.docstrings :
.... if toktype == tokenize.STRING and len(toktext) >= 6:
.... t = toktext.lstrip( 'rRuU')
.... if ((t.startswith( "'''") and t.endswith("''' ")) or
.... (t.startswith(' """') and t.endswith('""" '))):
.... return
[color=blue]
> # handle newlines
> if toktype in [token.NEWLINE, tokenize.NL]:
> self.temp.write (self.lineend)
> return
>
> # send the original whitespace, if needed
> if newpos > oldpos:
> self.temp.write (self.raw[oldpos:newpos])
>
> # skip indenting tokens
> if toktype in [token.INDENT, token.DEDENT]:
> self.pos = newpos
> return
>
> # send text to the temp file
> self.temp.write (toktext)
> return
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> def Main():
> import sys
> if sys.argv[1]:
> filein = open(sys.argv[1]).read()
> Stripper(filein ).format(out=sy s.stdout, comments=1,[/color]
untabify=1,[color=blue]
> eol='win')
>
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> if __name__ == '__main__':
> Main()
>
> M.E.Farmer[/color]

[qwweeeit]

Re: whitespace , comment stripper, and EOL converter

Hi,

Importing a text file from another o.s. is not a problem : I convert
it immediately using the powerful shell functions of Linux (and Unix).

I thank you for the explanation about classes, but I am rather dumb
and
by now I resolved all my problems without them...
Speaking of problems..., I have yet an error in parsing for literal
strings,
when there is more than one literal string by source line.

Perhaps it's time to use classes...

[M.E.Farmer]

Re: whitespace , comment stripper, and EOL converter

Thanks Jean,
I have thought about adding docstrings several times, but I was stumped
at how to determine a docstring from a regular tripleqoted string ;)
I have been thinking hard about the problem and I think I have an idea.
If the line has nothing before the start of the string it must be a
docstring.
Sounds simple enough but in Python there are 12 or so 'types' of
strings .
Here is my crack at it feel free to improve it ;)
I reversed the logic on the comments and docstrings so I could add a
special mode to docstring stripping ...pep8 mode .
Pep8 mode only strips double triple quotes from your source code
leaving the offending single triple quotes behind. Probably just stupid
but someone might find it usefull.
############### ############### ############### ############### ##########
# Python source stripper
############### ############### ############### ############### ##########

import os
import sys
import token
import keyword
import StringIO
import tokenize
import traceback
__credits__ = '''
Jürgen Hermann
M.E.Farmer
Jean Brouwers
'''
__version__ = '.8'
__author__ = 'M.E.Farmer'
__date__ = 'Apr 16, 2005,' \
'Jan 15 2005,' \
'Oct 24 2004' \


############### ############### ############### ############### ##########

class Stripper:
"""Python source stripper
"""
def __init__(self, raw):
self.raw = raw

def format(self, out=sys.stdout, comments=0, docstrings=0,
spaces=1, untabify=1, eol='unix'):
""" strip comments,
strip docstrings,
strip extra whitespace and lines,
convert tabs to spaces,
convert EOL's in Python code.
"""
# Store line offsets in self.lines
self.lines = [0, 0]
pos = 0
# Strips the first blank line if 1
self.lasttoken = 1
self.temp = StringIO.String IO()
self.spaces = spaces
self.comments = comments
self.docstrings = docstrings

if untabify:
self.raw = self.raw.expand tabs()
self.raw = self.raw.rstrip ()+' '
self.out = out

# Have you ever had a multiple line ending script?
# They can be nasty so lets get them all the same.
self.raw = self.raw.replac e('\r\n', '\n')
self.raw = self.raw.replac e('\r', '\n')
self.lineend = '\n'

# Gather lines
while 1:
pos = self.raw.find(s elf.lineend, pos) + 1
if not pos: break
self.lines.appe nd(pos)

self.lines.appe nd(len(self.raw ))
self.pos = 0

# Wrap text in a filelike object
text = StringIO.String IO(self.raw)

# Parse the source.
## Tokenize calls the __call__
## method for each token till done.
try:
tokenize.tokeni ze(text.readlin e, self)
except tokenize.TokenE rror, ex:
traceback.print _exc()

# Ok now we write it to a file
# but we also need to clean the whitespace
# between the lines and at the ends.
self.temp.seek( 0)

# All this should be written into the
# __call__ method just haven't yet...

# Mac CR
if eol == 'mac':
self.lineend = '\r'
# Windows CR LF
elif eol == 'win':
self.lineend = '\r\n'
# Unix LF
else:
self.lineend = '\n'

for line in self.temp.readl ines():
if spaces == -1:
self.out.write( line.rstrip()+s elf.lineend)
else:
if not line.isspace():
self.lasttoken= 0
self.out.write( line.rstrip()+s elf.lineend)
else:
self.lasttoken+ =1
if self.lasttoken< =self.spaces and self.spaces:
self.out.write( self.lineend)

def __call__(self, toktype, toktext,
(srow,scol), (erow,ecol), line):
""" Token handler.
"""
# calculate new positions
oldpos = self.pos
newpos = self.lines[srow] + scol
self.pos = newpos + len(toktext)

# kill comments
if self.comments:
if toktype == tokenize.COMMEN T:
return

# kill doc strings
if self.docstrings :
# Assume if there is nothing on the
# left side it must be a docstring
if toktype == tokenize.STRING and \
line.lstrip(' rRuU')[0] in ["'",'"']:
t = toktext.lstrip( 'rRuU')
if (t.startswith(' """') and
(self.docstring s == 'pep8' or
self.docstrings =='8')):
return
elif t.startswith('" ""') or t.startswith("' ''"):
return

# handle newlines
if toktype in [token.NEWLINE, tokenize.NL]:
self.temp.write (self.lineend)
return

# send the original whitespace
if newpos > oldpos:
self.temp.write (self.raw[oldpos:newpos])

# skip indenting tokens
if toktype in [token.INDENT, token.DEDENT]:
self.pos = newpos
return

# send text to the temp file
self.temp.write (toktext)
return
############### ############### ############### ############### ##########

def Main():
import sys
if sys.argv[1]:
filein = open(sys.argv[1]).read()
Stripper(filein ).format(out=sy s.stdout,
comments=0, docstrings=1, untabify=1, eol='win')
############### ############### ############### ############### ##########

if __name__ == '__main__':
Main()

[MrJean1]

Re: whitespace , comment stripper, and EOL converter

There is an issue with both my and your code: it only works if doc
strings are triple quoted and if there are no other triple quoted
strings in the Python code.

A triple quoted string used in an assignment will be removed, for
example this case

s = '''this string should not be removed'''


It is still unclear how to distinguish doc strings from other strings.
Also, I have not checked the precise Python syntax, but doc strings do
not need to be enclosed by triple quotes. A single quote may be
allowed too.

Maybe this rule will work: a doc string is any string preceded by a
COLON token followed by zero, one or more INDENT or NEWLINE tokens.
Untested!

/Jean Brouwers



M.E.Farmer wrote:[color=blue]
> Thanks Jean,
> I have thought about adding docstrings several times, but I was[/color]
stumped[color=blue]
> at how to determine a docstring from a regular tripleqoted string ;)
> I have been thinking hard about the problem and I think I have an[/color]
idea.[color=blue]
> If the line has nothing before the start of the string it must be a
> docstring.
> Sounds simple enough but in Python there are 12 or so 'types' of
> strings .
> Here is my crack at it feel free to improve it ;)
> I reversed the logic on the comments and docstrings so I could add a
> special mode to docstring stripping ...pep8 mode .
> Pep8 mode only strips double triple quotes from your source code
> leaving the offending single triple quotes behind. Probably just[/color]
stupid[color=blue]
> but someone might find it usefull.
>[/color]
############### ############### ############### ############### ##########[color=blue]
> # Python source stripper
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> import os
> import sys
> import token
> import keyword
> import StringIO
> import tokenize
> import traceback
> __credits__ = '''
> Jürgen Hermann
> M.E.Farmer
> Jean Brouwers
> '''
> __version__ = '.8'
> __author__ = 'M.E.Farmer'
> __date__ = 'Apr 16, 2005,' \
> 'Jan 15 2005,' \
> 'Oct 24 2004' \
>
>
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> class Stripper:
> """Python source stripper
> """
> def __init__(self, raw):
> self.raw = raw
>
> def format(self, out=sys.stdout, comments=0, docstrings=0,
> spaces=1, untabify=1, eol='unix'):
> """ strip comments,
> strip docstrings,
> strip extra whitespace and lines,
> convert tabs to spaces,
> convert EOL's in Python code.
> """
> # Store line offsets in self.lines
> self.lines = [0, 0]
> pos = 0
> # Strips the first blank line if 1
> self.lasttoken = 1
> self.temp = StringIO.String IO()
> self.spaces = spaces
> self.comments = comments
> self.docstrings = docstrings
>
> if untabify:
> self.raw = self.raw.expand tabs()
> self.raw = self.raw.rstrip ()+' '
> self.out = out
>
> # Have you ever had a multiple line ending script?
> # They can be nasty so lets get them all the same.
> self.raw = self.raw.replac e('\r\n', '\n')
> self.raw = self.raw.replac e('\r', '\n')
> self.lineend = '\n'
>
> # Gather lines
> while 1:
> pos = self.raw.find(s elf.lineend, pos) + 1
> if not pos: break
> self.lines.appe nd(pos)
>
> self.lines.appe nd(len(self.raw ))
> self.pos = 0
>
> # Wrap text in a filelike object
> text = StringIO.String IO(self.raw)
>
> # Parse the source.
> ## Tokenize calls the __call__
> ## method for each token till done.
> try:
> tokenize.tokeni ze(text.readlin e, self)
> except tokenize.TokenE rror, ex:
> traceback.print _exc()
>
> # Ok now we write it to a file
> # but we also need to clean the whitespace
> # between the lines and at the ends.
> self.temp.seek( 0)
>
> # All this should be written into the
> # __call__ method just haven't yet...
>
> # Mac CR
> if eol == 'mac':
> self.lineend = '\r'
> # Windows CR LF
> elif eol == 'win':
> self.lineend = '\r\n'
> # Unix LF
> else:
> self.lineend = '\n'
>
> for line in self.temp.readl ines():
> if spaces == -1:
> self.out.write( line.rstrip()+s elf.lineend)
> else:
> if not line.isspace():
> self.lasttoken= 0
> self.out.write( line.rstrip()+s elf.lineend)
> else:
> self.lasttoken+ =1
> if self.lasttoken< =self.spaces and self.spaces:
> self.out.write( self.lineend)
>
> def __call__(self, toktype, toktext,
> (srow,scol), (erow,ecol), line):
> """ Token handler.
> """
> # calculate new positions
> oldpos = self.pos
> newpos = self.lines[srow] + scol
> self.pos = newpos + len(toktext)
>
> # kill comments
> if self.comments:
> if toktype == tokenize.COMMEN T:
> return
>
> # kill doc strings
> if self.docstrings :
> # Assume if there is nothing on the
> # left side it must be a docstring
> if toktype == tokenize.STRING and \
> line.lstrip(' rRuU')[0] in ["'",'"']:
> t = toktext.lstrip( 'rRuU')
> if (t.startswith(' """') and
> (self.docstring s == 'pep8' or
> self.docstrings =='8')):
> return
> elif t.startswith('" ""') or t.startswith("' ''"):
> return
>
> # handle newlines
> if toktype in [token.NEWLINE, tokenize.NL]:
> self.temp.write (self.lineend)
> return
>
> # send the original whitespace
> if newpos > oldpos:
> self.temp.write (self.raw[oldpos:newpos])
>
> # skip indenting tokens
> if toktype in [token.INDENT, token.DEDENT]:
> self.pos = newpos
> return
>
> # send text to the temp file
> self.temp.write (toktext)
> return
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> def Main():
> import sys
> if sys.argv[1]:
> filein = open(sys.argv[1]).read()
> Stripper(filein ).format(out=sy s.stdout,
> comments=0, docstrings=1, untabify=1, eol='win')
>[/color]
############### ############### ############### ############### ##########[color=blue]
>
> if __name__ == '__main__':
> Main()[/color]

[M.E.Farmer]

Re: whitespace , comment stripper, and EOL converter

MrJean1 wrote:[color=blue]
> There is an issue with both my and your code: it only works if doc
> strings are triple quoted and if there are no other triple quoted
> strings in the Python code.[/color]
I had not considered single quoted strings ;)[color=blue]
> A triple quoted string used in an assignment will be removed, for
> example this case
>
> s = '''this string should not be removed'''
>
>
> It is still unclear how to distinguish doc strings from other[/color]
strings.[color=blue]
> Also, I have not checked the precise Python syntax, but doc strings[/color]
do[color=blue]
> not need to be enclosed by triple quotes. A single quote may be
> allowed too.
>
> Maybe this rule will work: a doc string is any string preceded by a
> COLON token followed by zero, one or more INDENT or NEWLINE tokens.
> Untested![/color]
Not needed , if you reread my post I explain that I had solved that
issue.
If you use the line argument that tokenizer supplies we can strip
whitespace and 'rRuU' from the start of the line and look for a single
quote or a double quote .
I have tested it and it works.
Reworked the 'pep8' thing and fixed the bug you mentioned here is the
changes.
[color=blue]
>[/color]
############### ############### ############### ############### ##########[color=blue][color=green]
> > # Python source stripper
> >[/color]
>[/color]
############### ############### ############### ############### ##########[color=blue][color=green]
> >
> > import os
> > import sys
> > import token
> > import keyword
> > import StringIO
> > import tokenize
> > import traceback
> > __credits__ = '''
> > Jürgen Hermann
> > M.E.Farmer
> > Jean Brouwers
> > '''
> > __version__ = '.8'
> > __author__ = 'M.E.Farmer'
> > __date__ = 'Apr 16, 2005,' \
> > 'Jan 15 2005,' \
> > 'Oct 24 2004' \
> >
> >
> >[/color]
>[/color]
############### ############### ############### ############### ##########[color=blue][color=green]
> >
> > class Stripper:
> > """Python source stripper
> > """
> > def __init__(self, raw):
> > self.raw = raw
> >
> > def format(self, out=sys.stdout, comments=0, docstrings=0,
> > spaces=1, untabify=1, eol='unix'):
> > """ strip comments,
> > strip docstrings,
> > strip extra whitespace and lines,
> > convert tabs to spaces,
> > convert EOL's in Python code.
> > """
> > # Store line offsets in self.lines
> > self.lines = [0, 0]
> > pos = 0
> > # Strips the first blank line if 1
> > self.lasttoken = 1
> > self.temp = StringIO.String IO()
> > self.spaces = spaces
> > self.comments = comments
> > self.docstrings = docstrings
> >
> > if untabify:
> > self.raw = self.raw.expand tabs()
> > self.raw = self.raw.rstrip ()+' '
> > self.out = out
> >
> > # Have you ever had a multiple line ending script?
> > # They can be nasty so lets get them all the same.
> > self.raw = self.raw.replac e('\r\n', '\n')
> > self.raw = self.raw.replac e('\r', '\n')
> > self.lineend = '\n'
> >
> > # Gather lines
> > while 1:
> > pos = self.raw.find(s elf.lineend, pos) + 1
> > if not pos: break
> > self.lines.appe nd(pos)
> >
> > self.lines.appe nd(len(self.raw ))
> > self.pos = 0
> >
> > # Wrap text in a filelike object
> > text = StringIO.String IO(self.raw)
> >
> > # Parse the source.
> > ## Tokenize calls the __call__
> > ## method for each token till done.
> > try:
> > tokenize.tokeni ze(text.readlin e, self)
> > except tokenize.TokenE rror, ex:
> > traceback.print _exc()
> >
> > # Ok now we write it to a file
> > # but we also need to clean the whitespace
> > # between the lines and at the ends.
> > self.temp.seek( 0)
> >
> > # All this should be written into the
> > # __call__ method just haven't yet...
> >
> > # Mac CR
> > if eol == 'mac':
> > self.lineend = '\r'
> > # Windows CR LF
> > elif eol == 'win':
> > self.lineend = '\r\n'
> > # Unix LF
> > else:
> > self.lineend = '\n'
> >
> > for line in self.temp.readl ines():
> > if spaces == -1:
> > self.out.write( line.rstrip()+s elf.lineend)
> > else:
> > if not line.isspace():
> > self.lasttoken= 0
> > self.out.write( line.rstrip()+s elf.lineend)
> > else:
> > self.lasttoken+ =1
> > if self.lasttoken< =self.spaces and self.spaces:
> > self.out.write( self.lineend)
> >
> > def __call__(self, toktype, toktext,
> > (srow,scol), (erow,ecol), line):
> > """ Token handler.
> > """
> > # calculate new positions
> > oldpos = self.pos
> > newpos = self.lines[srow] + scol
> > self.pos = newpos + len(toktext)
> >
> > # kill comments
> > if self.comments:
> > if toktype == tokenize.COMMEN T:
> > return
> >[/color][/color]
# kill doc strings
if self.docstrings :
# Assume if there is nothing on the
# left side it must be a docstring
if toktype == tokenize.STRING and \
line.lstrip(' rRuU')[0] in ["'",'"']:
t = toktext.lstrip( 'rRuU')
# pep8 frowns on triple single quotes
if ( self.docstrings == 'pep8' or
self.docstrings == 8):
# pep8 frowns on single triples
if not t.startswith('" ""'):
return
else:
return

# handle newlines
if toktype in [token.NEWLINE, tokenize.NL]:
self.temp.write (self.lineend)
return

# send the original whitespace
if newpos > oldpos:
self.temp.write (self.raw[oldpos:newpos])

# skip indenting tokens
if toktype in [token.INDENT, token.DEDENT]:
self.pos = newpos
return

# send text to the temp file
self.temp.write (toktext)
return
############### ############### ############### ############### ##########

def Main():
import sys
if sys.argv[1]:
filein = open(sys.argv[1]).read()
Stripper(filein ).format(out=sy s.stdout,
comments=0, docstrings='pep 8', untabify=1, eol='win')
############### ############### ############### ############### ##########

if __name__ == '__main__':
Main()


That should work like a charm for all types of docstrings without
disturbing others strings.

M.E.Farmer

[M.E.Farmer]

Re: whitespace , comment stripper, and EOL converter

Google has now 'fixed' there whitespace issue and now has an auto-quote
issue argggh!

The script is located at:

http://bellsouthpwp.net/m/e/mefjr75/python/stripper.py

M.E.Farmer

[M.E.Farmer]

Re: whitespace , comment stripper, and EOL converter

I found the bug and hope I have squashed it.
Single and qouble quoted strings that were assignments and spanned
multilines using \ , were chopped after the first line.
example:
__date__ = 'Apr 16, 2005,' \
'Jan 15 2005,' \
'Oct 24 2004'
became:
__date__ = 'Apr 16, 2005,' \

Not good :(

tokenizer sends this as:
name
operator
string
string
string
newline

I added test for string assignments that end in \.
A flag is set and then all strings till a newline are ignored.
Also rearranged the script a little.
Maybe that will do it ...
Updates available at[color=blue]
> The script is located at:
> http://bellsouthpwp.net/m/e/mefjr75/python/stripper.py
>
> M.E.Farmer[/color]