advice : how do you iterate with an acc ?

[vd12005@yahoo.fr]

hello,

i'm wondering how people from here handle this, as i often encounter
something like:

acc = [] # accumulator ;)
for line in fileinput.input ():
if condition(line) :
if acc: #1
doSomething(acc ) #1
acc = []
else:
acc.append(line )
if acc: #2
doSomething(acc ) #2

BTW i am particularly annoyed by #1 and #2 as it is a reptition, and i
think it is quite error prone, how will you do it in a pythonic way ?

regards

[bonono@gmail.com]

Re: advice : how do you iterate with an acc ?


vd12005@yahoo.f r wrote:[color=blue]
> hello,
>
> i'm wondering how people from here handle this, as i often encounter
> something like:
>
> acc = [] # accumulator ;)
> for line in fileinput.input ():
> if condition(line) :
> if acc: #1
> doSomething(acc ) #1
> acc = []
> else:
> acc.append(line )
> if acc: #2
> doSomething(acc ) #2
>
> BTW i am particularly annoyed by #1 and #2 as it is a reptition, and i
> think it is quite error prone, how will you do it in a pythonic way ?
>[/color]
I think it is quite ok as the last "if acc:" is just an "end-of-stream"
implicit marker, whereas during the loop, you have explicit markers to
signal end/start of blocks. There is no unwanted variable introduced
and I don't see how it can be error prone.

This is one of the case I won't try to make it a one liner, because it
is already very natural :-)

[Bengt Richter]

Re: advice : how do you iterate with an acc ?

On 2 Dec 2005 17:08:02 -0800, bonono@gmail.co m wrote:
[color=blue]
>
>vd12005@yahoo. fr wrote:[color=green]
>> hello,
>>
>> i'm wondering how people from here handle this, as i often encounter
>> something like:
>>
>> acc = [] # accumulator ;)
>> for line in fileinput.input ():
>> if condition(line) :
>> if acc: #1
>> doSomething(acc ) #1
>> acc = []
>> else:
>> acc.append(line )
>> if acc: #2
>> doSomething(acc ) #2
>>
>> BTW i am particularly annoyed by #1 and #2 as it is a reptition, and i
>> think it is quite error prone, how will you do it in a pythonic way ?
>>[/color][/color]
It looks to me like itertools.group by could get you close to what you want,
e.g., (untested)

import itertools
for condresult, acciter in itertools.group by(fileinput.im put(), condition):
if not condresult:
dosomething(lis t(acciter)) # or dosomething(acc iter) if iterator is usable

IOW, groupy collects contiguous lines for which condition evaluates to a distinct
value. Assuming this is a funtion that returns only two distinct values (for true
and false, like True and False), then if I understand your program's logic, you
do nothing with the line(s) that actually satisfy the condition, you just trigger
on them as delimiters and want to process the nonempty groups of the other lines,
so the "if not condresult:" should select those. Groupby won't return an empty group AFAIK,
so you don't need to test for that. Also, you won't need the list call in list(acciter)
if your dosomething can accept an iterator instead of a list.

Regards,
Bengt Richter

[Dan Sommers]

Re: advice : how do you iterate with an acc ?

On 2 Dec 2005 16:45:38 -0800,
vd12005@yahoo.f r wrote:
[color=blue]
> hello,
> i'm wondering how people from here handle this, as i often encounter
> something like:[/color]
[color=blue]
> acc = [] # accumulator ;)
> for line in fileinput.input ():
> if condition(line) :
> if acc: #1
> doSomething(acc ) #1
> acc = []
> else:
> acc.append(line )
> if acc: #2
> doSomething(acc ) #2[/color]
[color=blue]
> BTW i am particularly annoyed by #1 and #2 as it is a reptition, and i
> think it is quite error prone, how will you do it in a pythonic way ?[/color]

If doSomething handled an empty list gracefully, then you would have
less repetition:

acc = []
for line in fileinput.input ():
if condition(line) :
doSomething(acc ) #1
acc = []
else:
acc.append(line )
doSomething(acc ) #2

If condition were simple enough and the file(s) small enough, perhaps
you could read the whole file at once and use split to separate the
pieces:

contents = file.read()
for acc in contents.split( "this is the delimiter line\n" ):
doSomething(acc .split("\n"))

(There are probably some strange cases of repeated delimiter lines or
delimiter lines occurring at the beginning or end of the file for which
the above code will not work. Caveat emptor.)

If condition were a little more complicated, perhaps re.split would
work.

Or maybe you could look at split and see what it does (since your code
is conceptually very similar to it).

OTOH, FWIW, your version is very clean and very readable and fits my
brain perfectly.

HTH,
Dan

--
Dan Sommers
<http://www.tombstoneze ro.net/dan/>

[bonono@gmail.com]

Re: advice : how do you iterate with an acc ?


Bengt Richter wrote:[color=blue]
> It looks to me like itertools.group by could get you close to what you want,
> e.g., (untested)[/color]
Ah, groupby. The generic string.split() equivalent. But the doc said
the input needs to be sorted.

[Jeffrey Schwab]

Re: advice : how do you iterate with an acc ?

vd12005@yahoo.f r wrote:[color=blue]
> hello,
>
> i'm wondering how people from here handle this, as i often encounter
> something like:
>
> acc = [] # accumulator ;)
> for line in fileinput.input ():
> if condition(line) :
> if acc: #1
> doSomething(acc ) #1
> acc = []
> else:
> acc.append(line )
> if acc: #2
> doSomething(acc ) #2
>
> BTW i am particularly annoyed by #1 and #2 as it is a reptition, and i
> think it is quite error prone, how will you do it in a pythonic way ?[/color]

Could you add a sentry to the end of your input? E.g.:

for line in fileinput.input () + line_that_match es_condition:

This way, you wouldn't need a separate check at the end.

[Ben Finney]

Re: advice : how do you iterate with an acc ?

vd12005@yahoo.f r wrote:[color=blue]
> acc = [] # accumulator ;)
> for line in fileinput.input ():
> if condition(line) :
> if acc: #1
> doSomething(acc ) #1
> acc = []
> else:
> acc.append(line )
> if acc: #2
> doSomething(acc ) #2[/color]

Looks like you'd be better off making an Accumulator that knows what
to do.
[color=blue][color=green][color=darkred]
>>> class Accumulator(lis t):[/color][/color][/color]
... def flush(self):
... if len(self):
... print "Flushing items: %s" % self
... del self[:]
...[color=blue][color=green][color=darkred]
>>> lines = [[/color][/color][/color]
... "spam", "eggs", "FLUSH",
... "beans", "rat", "FLUSH",
... "strawberry ",
... ][color=blue][color=green][color=darkred]
>>>
>>> acc = Accumulator()
>>> for line in lines:[/color][/color][/color]
... if line == 'FLUSH':
... acc.flush()
... else:
... acc.append(line )
...
Flushing items: ['spam', 'eggs']
Flushing items: ['beans', 'rat'][color=blue][color=green][color=darkred]
>>> acc.flush()[/color][/color][/color]
Flushing items: ['strawberry'][color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

--
\ "[W]e are still the first generation of users, and for all that |
`\ we may have invented the net, we still don't really get it." |
_o__) -- Douglas Adams |
Ben Finney

[Bengt Richter]

Re: advice : how do you iterate with an acc ?

On 2 Dec 2005 18:34:12 -0800, bonono@gmail.co m wrote:
[color=blue]
>
>Bengt Richter wrote:[color=green]
>> It looks to me like itertools.group by could get you close to what you want,
>> e.g., (untested)[/color]
>Ah, groupby. The generic string.split() equivalent. But the doc said
>the input needs to be sorted.
>[/color]
[color=blue][color=green][color=darkred]
>>> seq = [3,1,4,'t',0,3,4 ,2,'t',3,1,4]
>>> import itertools
>>> def condition(item) : return item=='t'[/color][/color][/color]
...[color=blue][color=green][color=darkred]
>>> def dosomething(it) : return 'doing something with %r'%list(it)[/color][/color][/color]
...[color=blue][color=green][color=darkred]
>>> for condresult, acciter in itertools.group by(seq, condition):[/color][/color][/color]
... if not condresult:
... dosomething(acc iter)
...
'doing something with [3, 1, 4]'
'doing something with [0, 3, 4, 2]'
'doing something with [3, 1, 4]'

I think the input only needs to be sorted if you a trying to group sorted subsequences of the input.
I.e., you can't get them extracted together unless the condition is satisfied for a contiguous group, which
only happens if the input is sorted. But AFAIK the grouping logic just scans and applies key condition
and returns iterators for the subsequences that yield the same key function result, along with that result.
So it's a general subsequence extractor. You just have to supply the key function to make the condition value
change when a group ends and a new one begins. And the value can be arbitrary, or just toggle beween two values, e.g.
[color=blue][color=green][color=darkred]
>>> for condresult, acciter in itertools.group by(range(20), lambda x:x%3==0 or x==5):[/color][/color][/color]
... print '%6s: %r'%(condresult , list(acciter))
...
True: [0]
False: [1, 2]
True: [3]
False: [4]
True: [5, 6]
False: [7, 8]
True: [9]
False: [10, 11]
True: [12]
False: [13, 14]
True: [15]
False: [16, 17]
True: [18]
False: [19]

or a condresult that stays the same in groups, but every group result is different:
[color=blue][color=green][color=darkred]
>>> for condresult, acciter in itertools.group by(range(20), lambda x:x//3):[/color][/color][/color]
... print '%6s: %r'%(condresult , list(acciter))
...
0: [0, 1, 2]
1: [3, 4, 5]
2: [6, 7, 8]
3: [9, 10, 11]
4: [12, 13, 14]
5: [15, 16, 17]
6: [18, 19]

Regards,
Bengt Richter

[bonono@gmail.com]

Re: advice : how do you iterate with an acc ?


Bengt Richter wrote:[color=blue]
> On 2 Dec 2005 18:34:12 -0800, bonono@gmail.co m wrote:
>[color=green]
> >
> >Bengt Richter wrote:[color=darkred]
> >> It looks to me like itertools.group by could get you close to what you want,
> >> e.g., (untested)[/color]
> >Ah, groupby. The generic string.split() equivalent. But the doc said
> >the input needs to be sorted.
> >[/color]
>[color=green][color=darkred]
> >>> seq = [3,1,4,'t',0,3,4 ,2,'t',3,1,4]
> >>> import itertools
> >>> def condition(item) : return item=='t'[/color][/color]
> ...[color=green][color=darkred]
> >>> def dosomething(it) : return 'doing something with %r'%list(it)[/color][/color]
> ...[color=green][color=darkred]
> >>> for condresult, acciter in itertools.group by(seq, condition):[/color][/color]
> ... if not condresult:
> ... dosomething(acc iter)
> ...
> 'doing something with [3, 1, 4]'
> 'doing something with [0, 3, 4, 2]'
> 'doing something with [3, 1, 4]'
>
> I think the input only needs to be sorted if you a trying to group sorted subsequences of the input.
> I.e., you can't get them extracted together unless the condition is satisfied for a contiguous group, which
> only happens if the input is sorted. But AFAIK the grouping logic just scans and applies key condition
> and returns iterators for the subsequences that yield the same key function result, along with that result.
> So it's a general subsequence extractor. You just have to supply the key function to make the condition value
> change when a group ends and a new one begins. And the value can be arbitrary, or just toggle beween two values, e.g.
>[color=green][color=darkred]
> >>> for condresult, acciter in itertools.group by(range(20), lambda x:x%3==0 or x==5):[/color][/color]
> ... print '%6s: %r'%(condresult , list(acciter))
> ...
> True: [0]
> False: [1, 2]
> True: [3]
> False: [4]
> True: [5, 6]
> False: [7, 8]
> True: [9]
> False: [10, 11]
> True: [12]
> False: [13, 14]
> True: [15]
> False: [16, 17]
> True: [18]
> False: [19]
>
> or a condresult that stays the same in groups, but every group result is different:
>[color=green][color=darkred]
> >>> for condresult, acciter in itertools.group by(range(20), lambda x:x//3):[/color][/color]
> ... print '%6s: %r'%(condresult , list(acciter))
> ...
> 0: [0, 1, 2]
> 1: [3, 4, 5]
> 2: [6, 7, 8]
> 3: [9, 10, 11]
> 4: [12, 13, 14]
> 5: [15, 16, 17]
> 6: [18, 19]
>[/color]
Thanks. So it basically has an internal state storing the last
"condition" result and if it flips(different ), a new group starts.

[Scott David Daniels]

Re: advice : how do you iterate with an acc ?

Jeffrey Schwab wrote:[color=blue]
> vd12005@yahoo.f r wrote:[color=green]
>> hello,
>>
>> .... i often encounter something like:
>>
>> acc = [] # accumulator ;)
>> for line in fileinput.input ():
>> if condition(line) :
>> if acc: #1
>> doSomething(acc ) #1
>> acc = []
>> else:
>> acc.append(line )
>> if acc: #2
>> doSomething(acc ) #2[/color]
>
> Could you add a sentry to the end of your input? E.g.:
> for line in fileinput.input () + line_that_match es_condition:
> This way, you wouldn't need a separate check at the end.[/color]

Check itertools for a good way to do this:

import itertools
SENTRY = 'something for which condition(SENTR Y) is True'

f = open(filename)
try:
for line in itertools.chain (f, [SENTRY]):
if condition(line) :
if acc:
doSomething(acc )
acc = []
else:
acc.append(line )
assert acc == []
finally:
f.close()


--Scott David Daniels
scott.daniels@a cm.org

[Bengt Richter]

Re: advice : how do you iterate with an acc ?

On 3 Dec 2005 03:28:19 -0800, bonono@gmail.co m wrote:
[color=blue]
>
>Bengt Richter wrote:[color=green]
>> On 2 Dec 2005 18:34:12 -0800, bonono@gmail.co m wrote:
>>[color=darkred]
>> >
>> >Bengt Richter wrote:
>> >> It looks to me like itertools.group by could get you close to what you want,
>> >> e.g., (untested)
>> >Ah, groupby. The generic string.split() equivalent. But the doc said
>> >the input needs to be sorted.
>> >[/color]
>>[color=darkred]
>> >>> seq = [3,1,4,'t',0,3,4 ,2,'t',3,1,4]
>> >>> import itertools
>> >>> def condition(item) : return item=='t'[/color]
>> ...[color=darkred]
>> >>> def dosomething(it) : return 'doing something with %r'%list(it)[/color]
>> ...[color=darkred]
>> >>> for condresult, acciter in itertools.group by(seq, condition):[/color]
>> ... if not condresult:
>> ... dosomething(acc iter)
>> ...
>> 'doing something with [3, 1, 4]'
>> 'doing something with [0, 3, 4, 2]'
>> 'doing something with [3, 1, 4]'
>>
>> I think the input only needs to be sorted if you a trying to group sorted subsequences of the input.
>> I.e., you can't get them extracted together unless the condition is satisfied for a contiguous group, which
>> only happens if the input is sorted. But AFAIK the grouping logic just scans and applies key condition
>> and returns iterators for the subsequences that yield the same key function result, along with that result.
>> So it's a general subsequence extractor. You just have to supply the key function to make the condition value
>> change when a group ends and a new one begins. And the value can be arbitrary, or just toggle beween two values, e.g.
>>[color=darkred]
>> >>> for condresult, acciter in itertools.group by(range(20), lambda x:x%3==0 or x==5):[/color]
>> ... print '%6s: %r'%(condresult , list(acciter))
>> ...
>> True: [0]
>> False: [1, 2]
>> True: [3]
>> False: [4]
>> True: [5, 6]
>> False: [7, 8]
>> True: [9]
>> False: [10, 11]
>> True: [12]
>> False: [13, 14]
>> True: [15]
>> False: [16, 17]
>> True: [18]
>> False: [19]
>>
>> or a condresult that stays the same in groups, but every group result is different:
>>[color=darkred]
>> >>> for condresult, acciter in itertools.group by(range(20), lambda x:x//3):[/color]
>> ... print '%6s: %r'%(condresult , list(acciter))
>> ...
>> 0: [0, 1, 2]
>> 1: [3, 4, 5]
>> 2: [6, 7, 8]
>> 3: [9, 10, 11]
>> 4: [12, 13, 14]
>> 5: [15, 16, 17]
>> 6: [18, 19]
>>[/color]
>Thanks. So it basically has an internal state storing the last
>"condition" result and if it flips(different ), a new group starts.
>[/color]
So it appears. But note that "flips(differen t)" seems to be based on ==,
and default key function is just passthrough like lambda x:x, so e.g. integers
and floats will group together if their values are equal.
E.g., to elucidate further,

Default key function:[color=blue][color=green][color=darkred]
>>> from itertools import groupby
>>> for k,g in groupby([0, 0.0, 0j, [], (), None, 1, 1.0, 1j]):[/color][/color][/color]
... print k, list(g)
...
0 [0, 0.0, 0j]
[] [[]]
() [()]
None [None]
1 [1, 1.0]
1j [1j]

Group by bool value:[color=blue][color=green][color=darkred]
>>> for k,g in groupby([0, 0.0, 0j, [], (), None, 1, 1.0, 1j], key=bool):[/color][/color][/color]
... print k, list(g)
...
False [0, 0.0, 0j, [], (), None]
True [1, 1.0, 1j]

It's not trying to sort, so it doesn't trip on complex[color=blue][color=green][color=darkred]
>>> for k,g in groupby([0, 0.0, 0j, [], (), None, 1, 1.0, 1j, 2j]):[/color][/color][/color]
... print k, list(g)
...
0 [0, 0.0, 0j]
[] [[]]
() [()]
None [None]
1 [1, 1.0]
1j [1j]
2j [2j]

But you have to watch out if you try to pre-sort stuff that includes complex numbers[color=blue][color=green][color=darkred]
>>> for k,g in groupby(sorted([0, 0.0, 0j, [], (), None, 1, 1.0, 1j, 2j])):[/color][/color][/color]
... print k, list(g)
...
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: cannot compare complex numbers using <, <=, >, >=

And if you do sort using a key function, it doesn't mean groupy inherits that keyfunction for grouping
unless you specify it
[color=blue][color=green][color=darkred]
>>> def keyfun(x):[/color][/color][/color]
... if isinstance(x, (int, long, float)): return x
... else: return type(x).__name_ _
...[color=blue][color=green][color=darkred]
>>> for k,g in groupby(sorted([0, 0.0, 0j, [], (), None, 1, 1.0, 1j, 2j], key=keyfun)):[/color][/color][/color]
... print k, list(g)
...
0 [0, 0.0]
1 [1, 1.0]
None [None]
0j [0j]
1j [1j]
2j [2j]
[] [[]]
() [()]

Vs giving groupby the same keyfun[color=blue][color=green][color=darkred]
>>> for k,g in groupby(sorted([0, 0.0, 0j, [], (), None, 1, 1.0, 1j, 2j], key=keyfun), keyfun):[/color][/color][/color]
... print k, list(g)
...
0 [0, 0.0]
1 [1, 1.0]
NoneType [None]
complex [0j, 1j, 2j]
list [[]]
tuple [()]


Exmple of unsorted vs sorted subgroup extraction:
[color=blue][color=green][color=darkred]
>>> for k,g in groupby('this that other thing note order'.split(), key=lambda s:s[0]):[/color][/color][/color]
... print k, list(g)
...
t ['this', 'that']
o ['other']
t ['thing']
n ['note']
o ['order']

vs.
[color=blue][color=green][color=darkred]
>>> for k,g in groupby(sorted( 'this that other thing note order'.split()) , key=lambda s:s[0]):[/color][/color][/color]
... print k, list(g)
...
n ['note']
o ['order', 'other']
t ['that', 'thing', 'this']

Oops, that key would be less brittle as (untested) key=lambda s:s[:1], e.g., in case a split with args was used.

Regards,
Bengt Richter