for line in file weirdness

[Cordula's Web]

Hello,

here's a strange bug (?) I've came across (using Python 2.2):

# loop_1
for line in file:
if some_condition( line): break
do_something()

# loop_2
for line in file:
do_something_el se()

The problem is, that loop_2 doesn't resume where loop_1 left off, but
skips many lines (a block's worth or so) before continuing.

Why is this? Is reading from a file non-reentrant?

It is always possible to slurp the whole file content into a list, and
then iterate through the list, but I want to handle HUGE files too.

Thanks,
-cpghost.

--
Cordula's Web. http://www.cordula.ws/

[Fredrik Lundh]

Re: for line in file weirdness

"Cordula's Web" <cpghost@cordul a.ws> wrote:
[color=blue]
> here's a strange bug (?) I've came across (using Python 2.2):
>
> # loop_1
> for line in file:
> if some_condition( line): break
> do_something()
>
> # loop_2
> for line in file:
> do_something_el se()
>
> The problem is, that loop_2 doesn't resume where loop_1 left off, but
> skips many lines (a block's worth or so) before continuing.
>
> Why is this? Is reading from a file non-reentrant?[/color]

as mentioned in the documentation, the iterator interface (which is used by the
for-in machiner) uses a read-ahead buffer. in 2.2, whenever you enter a new
loop, a new read-ahead buffer is created, and it starts where the last one ended,
rather than right after the last line you read.

to get more reliable results in 2.2, you can create the iterator outside the loop,
and loop over the iterator object instead of the file itself.

file = iter(open(...))
for line in file:
if some_condition( line): break
do_something()
for line in file:
do_something_el se()

(iirc, this quirk was fixed in 2.3)

</F>

[SamIam]

Re: for line in file weirdness

I think what you need to do is to have a nested if_else statment:
for line in filelines:
if some_condition : break
else: do_something_el se

If the if statment is excuted then break return to for_loop
else do something different then return to for_loop.
When I read from a file I read the whole file into a variable then
work form the variable

file = open('InputStri ng','r') # open file
for reading only
filelines = map(string.stri p,file.readline s()) #remove newlines
for string

Then you can just use the variable filelines and loop through as much
as you like. If I can help you can email me at servando@mac.co m
I also use SKYPE username servando_garcia
Hope this helped.

Cordula's Web wrote:[color=blue]
> Hello,
>
> here's a strange bug (?) I've came across (using Python 2.2):
>
> # loop_1
> for line in file:
> if some_condition( line): break
> do_something()
>
> # loop_2
> for line in file:
> do_something_el se()
>
> The problem is, that loop_2 doesn't resume where loop_1 left off, but
> skips many lines (a block's worth or so) before continuing.
>
> Why is this? Is reading from a file non-reentrant?
>
> It is always possible to slurp the whole file content into a list,[/color]
and[color=blue]
> then iterate through the list, but I want to handle HUGE files too.
>
> Thanks,
> -cpghost.
>
> --
> Cordula's Web. http://www.cordula.ws/[/color]

[Cordula's Web]

Re: for line in file weirdness

A read-ahead buffer? Yes, that would explain it. Sorry, I missed this
piece of information in the documentation.

Thanks to all who replied.

[Cordula's Web]

Re: for line in file weirdness

Thanks :)

Reading everything into a variable was not an option, due to some very
large files. Creating the iterator only once, as Fredrik suggested,
solved the problem nicely.

Again many thanks for your great support!