Is there a better/simpler way to filter blank lines?

[tmallen]

I'm parsing some text files, and I want to strip blank lines in the
process. Is there a simpler way to do this than what I have here?

lines = filter(lambda line: len(line.strip( )) 0, lines)

Thomas

[bearophileHUGS@lycos.com]

Re: Is there a better/simpler way to filter blank lines?

tmallen:

I'm parsing some text files, and I want to strip blank lines in the
process. Is there a simpler way to do this than what I have here?
lines = filter(lambda line: len(line.strip( )) 0, lines)

xlines = (line for line in open(filename) if line.strip())

Bye,
bearophile

[Larry Bates]

Re: Is there a better/simpler way to filter blank lines?

bearophileHUGS@ lycos.com wrote:

tmallen:

>I'm parsing some text files, and I want to strip blank lines in the
>process. Is there a simpler way to do this than what I have here?
>lines = filter(lambda line: len(line.strip( )) 0, lines)

>
xlines = (line for line in open(filename) if line.strip())
>
Bye,
bearophile

Of if you want to filter/loop at the same time, or if you don't want all the
lines in memory at the same time:

fp = open(filename, 'r')
for line in fp:
if not line.strip():
continue

#
# Do something with the non-blank like:
#


fp.close()

-Larry

[tmallen]

Re: Is there a better/simpler way to filter blank lines?

On Nov 4, 4:30 pm, bearophileH...@ lycos.com wrote:

tmallen:
>

I'm parsing some text files, and I want to strip blank lines in the
process. Is there a simpler way to do this than what I have here?
lines = filter(lambda line: len(line.strip( )) 0, lines)

>
xlines = (line for line in open(filename) if line.strip())
>
Bye,
bearophile

I must be missing something:

>>xlines = (line for line in open("new.data" ) if line.strip())
>>xlines

<generator object at 0x6b648>

>>xlines.sort ()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'generator' object has no attribute 'sort'

What do you think?

Thomas

[Steven D'Aprano]

Re: Is there a better/simpler way to filter blank lines?

On Tue, 04 Nov 2008 13:27:00 -0800, tmallen wrote:

I'm parsing some text files, and I want to strip blank lines in the
process. Is there a simpler way to do this than what I have here?
>
lines = filter(lambda line: len(line.strip( )) 0, lines)
>
Thomas

lines = filter(lambda line: line.strip(), lines)


--
Steven

[Chris Rebert]

Re: Is there a better/simpler way to filter blank lines?

On Tue, Nov 4, 2008 at 2:30 PM, tmallen <thomasmallen@g mail.comwrote:

On Nov 4, 4:30 pm, bearophileH...@ lycos.com wrote:

>tmallen:
>>

I'm parsing some text files, and I want to strip blank lines in the
process. Is there a simpler way to do this than what I have here?
lines = filter(lambda line: len(line.strip( )) 0, lines)

>>
>xlines = (line for line in open(filename) if line.strip())
>>
>Bye,
>bearophile

>
I must be missing something:
>

>>>xlines = (line for line in open("new.data" ) if line.strip())
>>>xlines

<generator object at 0x6b648>

>>>xlines.sort( )

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'generator' object has no attribute 'sort'
>
What do you think?

xlines is a generator, not a list. If you don't know what a generator
is, see the relevant parts of the Python tutorial/manual (Google is
your friend).
To sort the generator, you can use 'sorted(xlines) '
If you need it to actually be a list, you can do 'list(xlines)'

Cheers,
Chris
--
Follow the path of the Iguana...

http://rebertia.com

>
Thomas
--

http://mail.python.org/mailman/listinfo/python-list

>

[Ben Finney]

Re: Is there a better/simpler way to filter blank lines?

Larry Bates <larry.bates@vi talEsafe.comwri tes:

bearophileHUGS@ lycos.com wrote:

xlines = (line for line in open(filename) if line.strip())

>
Of if you want to filter/loop at the same time, or if you don't want
all the lines in memory at the same time

The above implementation creates a generator; so it, too, won't need
to load all the lines in memory at the same time

--
\ “Program testing can be a very effective way to show the |
`\ presence of bugs, but is hopelessly inadequate for showing |
_o__) their absence.” —Edsger W. Dijkstra |
Ben Finney

[Ben Finney]

Re: Is there a better/simpler way to filter blank lines?

tmallen <thomasmallen@g mail.comwrites:

On Nov 4, 4:30 pm, bearophileH...@ lycos.com wrote:

xlines = (line for line in open(filename) if line.strip())

>
I must be missing something:
>

>xlines = (line for line in open("new.data" ) if line.strip())
>xlines

<generator object at 0x6b648>

A generator <URL:http://www.python.org/dev/peps/pep-0255is a
sequence, but is not a collection. It will generate each item on
request, rather than having them all in memory at once.

for line in xlines:
do something_knowi ng_the_line_is_ not_blank(line)

If you later *want* a collection containing all the items from the
generator, you can feed the generator (or any iterable) to a type that
can turn it into a collection. For example, to get all the filtered
lines as a list:

all_lines = list(xlines)

Note that some generators (not this one, which will end because the
file is finite size) never end, so feeding them to a constructor this
way will never return.

--
\ “It is far better to grasp the universe as it really is than to |
`\ persist in delusion, however satisfying and reassuring.” —Carl |
_o__) Sagan |
Ben Finney

[bearophileHUGS@lycos.com]

Re: Is there a better/simpler way to filter blank lines?

tmallen

I must be missing something:
>

>xlines = (line for line in open("new.data" ) if line.strip())
>xlines

<generator object at 0x6b648>

>xlines.sort( )

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'generator' object has no attribute 'sort'
>
What do you think?

Congratulations , you have just met your first lazy construct ^_^
That's a generator, it yields nonblank lines one after the other. This
can be really useful.
If you want a real array of items, then you can do this:
lines = list(xlines)
Or use a list comp.:
lines = [line for line in open("new.data" ) if line.strip()]

Bye,
bearophile

[Falcolas]

Re: Is there a better/simpler way to filter blank lines?

On Nov 4, 3:30 pm, tmallen <thomasmal...@g mail.comwrote:

On Nov 4, 4:30 pm, bearophileH...@ lycos.com wrote:
>

tmallen:

>

I'm parsing some text files, and I want to strip blank lines in the
process. Is there a simpler way to do this than what I have here?
lines = filter(lambda line: len(line.strip( )) 0, lines)

>

xlines = (line for line in open(filename) if line.strip())

>

Bye,
bearophile

>
I must be missing something:
>

>xlines = (line for line in open("new.data" ) if line.strip())
>xlines

>
<generator object at 0x6b648>>>xline s.sort()
>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'generator' object has no attribute 'sort'
>
What do you think?
>
Thomas

Using the surrounding parentheses creates a generator object, whereas
using square brackets would create a list. So, if you want to run list
operations on the resulting object, you'll want to use the list
comprehension instead.

i.e.

list_o_lines = [line for line in open(filename) if line.strip()]

Downside is the increased memory usage and processing time as you dump
the entire file into memory, whereas if you plan to do a "for line in
xlines:" operation, it would be faster to use the generator.

[tmallen]

Re: Is there a better/simpler way to filter blank lines?

Between this info and http://www.python.org/doc/2.5.2/tut/...00000000000000
, I'm starting to understand how I'll use generators (I've seen them
mentioned before, but never used them knowingly).

list_o_lines = [line for line in open(filename) if line.strip()]

+1 for "list_o_lin es"

Thanks for the help!
Thomas

On Nov 4, 6:36 pm, Falcolas <garri...@gmail .comwrote:

On Nov 4, 3:30 pm, tmallen <thomasmal...@g mail.comwrote:
>
>
>

On Nov 4, 4:30 pm, bearophileH...@ lycos.com wrote:

>

tmallen:

>

I'm parsing some text files, and I want to strip blank lines in the
process. Is there a simpler way to do this than what I have here?
lines = filter(lambda line: len(line.strip( )) 0, lines)

>

xlines = (line for line in open(filename) if line.strip())

>

Bye,
bearophile

>

I must be missing something:

>

>>xlines = (line for line in open("new.data" ) if line.strip())
>>xlines

>

<generator object at 0x6b648>>>xline s.sort()

>

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'generator' object has no attribute 'sort'

>

What do you think?

>

Thomas

>
Using the surrounding parentheses creates a generator object, whereas
using square brackets would create a list. So, if you want to run list
operations on the resulting object, you'll want to use the list
comprehension instead.
>
i.e.
>
list_o_lines = [line for line in open(filename) if line.strip()]
>
Downside is the increased memory usage and processing time as you dump
the entire file into memory, whereas if you plan to do a "for line in
xlines:" operation, it would be faster to use the generator.

[Ben Finney]

Re: Is there a better/simpler way to filter blank lines?

Falcolas <garrickp@gmail .comwrites:

Using the surrounding parentheses creates a generator object

No. Using the generator expression syntax creates a generator object.

Parentheses are irrelevant to whether the expression is a generator
expression. The parentheses merely group the expression from
surrounding syntax.

--
\ “bash awk grep perl sed, df du, du-du du-du, vi troff su fsck |
`\ rm * halt LART LART LART!” —The Swedish BOFH, |
_o__) alt.sysadmin.re covery |
Ben Finney

[Steve Holden]

Re: Is there a better/simpler way to filter blank lines?

tmallen wrote:

On Nov 4, 4:30 pm, bearophileH...@ lycos.com wrote:

>tmallen:
>>

>>I'm parsing some text files, and I want to strip blank lines in the
>>process. Is there a simpler way to do this than what I have here?
>>lines = filter(lambda line: len(line.strip( )) 0, lines)

>xlines = (line for line in open(filename) if line.strip())
>>
>Bye,
>bearophile

>
I must be missing something:
>

>>>xlines = (line for line in open("new.data" ) if line.strip())
>>>xlines

<generator object at 0x6b648>

>>>xlines.sort( )

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'generator' object has no attribute 'sort'
>
What do you think?
>

I think there'd be no advantage to a sort method on a generator, since
theoretically the last item could be the first required in the sorted
sequence, so it's necessary to hold all items in memory to ensure the
sort is correct. So there's no point using a generator in the first place.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[Marc 'BlackJack' Rintsch]

Re: Is there a better/simpler way to filter blank lines?

On Wed, 05 Nov 2008 12:06:42 +1100, Ben Finney wrote:

Falcolas <garrickp@gmail .comwrites:
>

>Using the surrounding parentheses creates a generator object

>
No. Using the generator expression syntax creates a generator object.
>
Parentheses are irrelevant to whether the expression is a generator
expression. The parentheses merely group the expression from surrounding
syntax.

No they are important:

In [270]: a = x for x in xrange(10)
------------------------------------------------------------
File "<ipython console>", line 1
a = x for x in xrange(10)
^
<type 'exceptions.Syn taxError'>: invalid syntax


In [271]: a = (x for x in xrange(10))

Ciao,
Marc 'BlackJack' Rintsch

[Ben Finney]

Re: Is there a better/simpler way to filter blank lines?

Marc 'BlackJack' Rintsch <bj_666@gmx.net writes:

On Wed, 05 Nov 2008 12:06:42 +1100, Ben Finney wrote:
>

Falcolas <garrickp@gmail .comwrites:

Using the surrounding parentheses creates a generator object

No. Using the generator expression syntax creates a generator
object.

Parentheses are irrelevant to whether the expression is a
generator expression. The parentheses merely group the expression
from surrounding syntax.

>
No they are important:

Your example shows only that they're important for grouping the
expression from surrounding syntax. As I said.

They are *not* important for making the expresison be a generator
expression in the first place. Parentheses are irrelevant for the
generator expression syntax.

--
\ “Today, I was — no, that wasn't me.” —Steven Wright |
`\ |
_o__) |
Ben Finney