Python Generators

[mpc]

HI all,
I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

#!/usr/bin/env python
import os,struct,time
def generate_header s(cachedir):
for name, blocksize in cachefiles:
pathname = os.path.join(ca chedir,name)
f = open(pathname," rb")
f.seek(4096)
while True:
header = f.read(36)
if not header: break
fields = struct.unpack(" >9I",header)
if fields[0] == 0x00010008: yield f, fields
fp = f.tell()
offset = fp % blocksize
if offset:
f.seek(blocksiz e - offset,1)
f.close()
def concatenate(seq uences):
for seq in sequences:
for item in seq:
yield item

all_caches = (path for path,dirlist,fi lelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
cachefiles = [('_CACHE_001_', 256),('_CACHE_0 02_',1024),('_C ACHE_003_',
4096)]
n = 0
while True:
n += 1
time.sleep(0.5)
headers = concatenate(gen erate_headers(p ath) for path in
all_caches)
for h in headers:
print h,n
# this doesn't work either

while True:
n += 1
time.sleep(0.5)
for path in all_caches:
headers = generate_header s(path)
for h in headers:
print h,n
# however if I hard code the path
path = "FFCache"
while True:
n += 1
headers = generate_header s(path)
for h in headers:
print h,n
#but of course i do not wish to hard code the path.

[sturlamolden]

Re: Python Generators

On 15 Mar, 21:35, mpc <mcoh...@gmail. comwrote:

generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

I am not sure what you are trying to do, but it seems a bit confused.

>>def concat(seq):

for s in seq: yield s

>>seq = xrange(3)

>>for n in xrange(5):

h = concat(s for s in seq)
for i in h: print i,n


0 0
1 0
2 0
0 1
1 1
2 1
0 2
1 2
2 2
0 3
1 3
2 3
0 4
1 4
2 4

Generators work they way they should. Even when one is used as
argument for another.

[Peter Otten]

Re: Python Generators

mpc wrote:

I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

A generator or generator expression does indeed only run once.

>>for i in range(3):

.... print "---", i, "---"
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---

--- 2 ---

The fix is to use a list or list comprehension, or make a new generator
every time you need one:

>>for i in range(3):

.... print "---", i, "---"
.... gen = (i for i in range(5) if i%2)
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---
1 3
--- 2 ---
1 3

At first glance I would guess that in your case all_caches is the culprit
that has to be moved into the 'while True: ...' loop.

while True:
n += 1
time.sleep(0.5)

all_caches = (path for path,dirlist,fi lelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)

for path in all_caches:
headers = generate_header s(path)
for h in headers:
print h,n

(Untested, because you didn't bother to provide a self-contained example)

Peter

[Peter Otten]

Re: Python Generators

mpc wrote:

I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

A generator or generator expression does indeed only run once.

>>gen = (i for i in range(5) if i%2)
>>for i in range(3):

.... print "---", i, "---"
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---

--- 2 ---

The fix is to use a list or list comprehension, or make a new generator
every time you need one:

>>for i in range(3):

.... print "---", i, "---"
.... gen = (i for i in range(5) if i%2)
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---
1 3
--- 2 ---
1 3

At first glance I would guess that in your case all_caches is the culprit
that has to be moved into the 'while True: ...' loop.

while True:
n += 1
time.sleep(0.5)

all_caches = (path for path,dirlist,fi lelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)

for path in all_caches:
headers = generate_header s(path)
for h in headers:
print h,n

(Untested, because you didn't bother to provide a self-contained example)

Peter

[Matt Nordhoff]

Re: Python Generators

mpc wrote:

<snip>

def concatenate(seq uences):
for seq in sequences:
for item in seq:
yield item

You should check out itertools.chain (). It does this. You call it like
"chain(seq1 , seq2, ...)" instead of "chain(sequence s)" though, which may
be a problem for you.

The rest of itertools might be interesting too:

<http://docs.python.org/lib/module-itertools.html>

<snip>
--