Beginner question - How to effectively pass a large list

[J.R.]

Hi folks,

The python can only support passing value in function call (right?), I'm
wondering how to effectively pass a large parameter, such as a large list or
dictionary?

It could achieved by pointer in C++, is there such way in Python?

Thansk in advance.
J.R.

[Bruno Desthuilliers]

Re: Beginner question - How to effectively pass a large list

J.R. wrote:[color=blue]
> Hi folks,
>
> The python can only support passing value in function call (right?),[/color]

let's see :[color=blue][color=green][color=darkred]
>>> l = range(5)
>>> def modif(alist): alist[0] = 'allo'[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> l[/color][/color][/color]
[0, 1, 2, 3, 4][color=blue][color=green][color=darkred]
>>> modif(l)
>>> l[/color][/color][/color]
['allo', 1, 2, 3, 4]

Er... Ok, let's try something else :

[color=blue][color=green][color=darkred]
>>> class Toto:[/color][/color][/color]
.... pass
....[color=blue][color=green][color=darkred]
>>> t = Toto()
>>> t.name = "Toto"
>>> def rename(t): t.name = "Titi"[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> t[/color][/color][/color]
<__main__.Tot o instance at 0x402dccec>[color=blue][color=green][color=darkred]
>>> t.name[/color][/color][/color]
'Toto'[color=blue][color=green][color=darkred]
>>> rename(t)
>>> t[/color][/color][/color]
<__main__.Tot o instance at 0x402dccec>[color=blue][color=green][color=darkred]
>>> t.name[/color][/color][/color]
'Titi'


Well... You may want to read more about bindings (not 'assignements') in
Python.

(hint : it already *does* work like pointers in C[++] - at least for
mutable objects).

Bruno

[Ben Finney]

Re: Beginner question - How to effectively pass a large list

On Mon, 15 Dec 2003 15:14:48 +0800, J.R. wrote:[color=blue]
> The python can only support passing value in function call (right?)[/color]

Wrong. Function parameters in Python are always passed by reference,
not by value. The local function parameter gets a binding to the same
object that was passed, so there are not two copies of the object in
memory.

Of course, because of the way that Python treats assignment, you can
then change that local parameter within the function and it will re-bind
to the new value, without altering the original value passed.

--
\ "I went to a fancy French restaurant called 'Déjà Vu'. The head |
`\ waiter said, 'Don't I know you?'" -- Steven Wright |
_o__) |
Ben Finney <http://bignose.squidly .org/>

[Donn Cave]

Re: Beginner question - How to effectively pass a large list

In article <3fde309f$0$192 85$626a54ce@new s.free.fr>,
Bruno Desthuilliers <bdesth.nospam@ removeme.free.f r> wrote:
....[color=blue]
> (hint : it already *does* work like pointers in C[++] - at least for
> mutable objects).[/color]

For any and all objects, irrespective of mutability.

Donn Cave, donn@u.washingt on.edu

[J.R.]

Re: Beginner question - How to effectively pass a large list

Thanks for the response.

I got following conclusion after reading your reply and other documents:

Actually, the python is passing the identity (i.e. memory address) of each
parameter, and it will bind to a local name within the function.

Right?

Thanks,
J.R.

"Donn Cave" <donn@u.washing ton.edu> wrote in message
news:donn-02CC52.16050615 122003@nntp3.u. washington.edu. ..[color=blue]
> In article <3fde309f$0$192 85$626a54ce@new s.free.fr>,
> Bruno Desthuilliers <bdesth.nospam@ removeme.free.f r> wrote:
> ...[color=green]
> > (hint : it already *does* work like pointers in C[++] - at least for
> > mutable objects).[/color]
>
> For any and all objects, irrespective of mutability.
>
> Donn Cave, donn@u.washingt on.edu[/color]

[Terry Reedy]

Re: Beginner question - How to effectively pass a large list


"J.R." <j.r.gao@motoro la.com> wrote in message
news:brmf9g$t59 $1@newshost.mot .com...[color=blue]
> I got following conclusion after reading your[/color]
reply and other documents:[color=blue]
>
> Actually, the python is passing the identity[/color]
(i.e. memory address) of each[color=blue]
> parameter, and it will bind to a local name[/color]
within the function.

Some months ago, there was a long thread on
whether Python function calling is 'call by name',
'call by value', or something else. Without
reiterating long discussion, I think from a user's
viewpoint, it is best considered 'call by
name-binding', with the value-carrying object
'passing' being by cross-namespace binding. One
can think of the return process as being similar
in that the return object is substituted for the
function name as if it had been bound to that
name.

The usage I believe promoted by Knuth and
supported by some here defines 'parameter' as the
within-function local name and 'argument' as the
outside value/object bound to that name at a
particular function call. Yes, CPython does that
with ids that are memory addresses, but that is
implementation rather than part of the language
definition itself. Who know what we do when we
act as Python interpreters!

Terry J. Reedy

[Donn Cave]

Re: Beginner question - How to effectively pass a large list

In article <brmf9g$t59$1@n ewshost.mot.com >,
"J.R." <j.r.gao@motoro la.com> wrote:
[color=blue]
> Thanks for the response.
>
> I got following conclusion after reading your reply and other documents:
>
> Actually, the python is passing the identity (i.e. memory address) of each
> parameter, and it will bind to a local name within the function.[/color]

That should work. This notion of binding a object to a name or
data structure is all over Python, as you have probably noticed,
and it will be time well spent if you experiment with it a little.

For example, how could you verify that arrays are passed without
copying? Unless you are unusually dense for a programmer or have
no access to a Python interpreter, that could be no more than a
couple of minutes work. As penance for having failed to do this,
I assign a more mysterious problem to you:

def f(d=[]):
d.append(0)
print d
f()
f()

Explain results. When is d bound?

Here's an easier one:

a = {'z': 0}
b = [a] * 4
b[0]['z'] = 1
print b

If the result is anywhere near a surprise to you, then you would
do well to stick to this line of inquiry if you want to do anything
more than the most trivial Python programming.

Donn Cave, donn@u.washingt on.edu

[J.R.]

Re: Beginner question - How to effectively pass a large list

> As penance for having failed to do this,[color=blue]
> I assign a more mysterious problem to you:
>
> def f(d=[]):
> d.append(0)
> print d
> f()
> f()
>
> Explain results. When is d bound?
>[/color]

I found that the "problem" showed above is caused by the python principle
"everything
is object".

Here the function "f" is an object as well, the object function f is created
once this function is defined. And there is tuple attribute(func_ defaults)
inside
function object to record the default arguments.[color=blue][color=green][color=darkred]
>>> def f(d=[]):[/color][/color][/color]
print id(d)
d.append(0)
print d[color=blue][color=green][color=darkred]
>>> f.func_defaults[/color][/color][/color]
([],)[color=blue][color=green][color=darkred]
>>> id(f.func_defau lts[0])[/color][/color][/color]
11279792[color=blue][color=green][color=darkred]
>>> f()[/color][/color][/color]
11279792
[0][color=blue][color=green][color=darkred]
>>> f()[/color][/color][/color]
11279792
[0, 0][color=blue][color=green][color=darkred]
>>> f([1])[/color][/color][/color]
11279952
[1, 0]

1. There is no value passed to the default argument
The name "d" is bound to the first element of the f.func_defaults . Since the
function "f" is an
object, which will be kept alive as long as there is name (current is "f")
refered to it, the
list in the func_defaults shall be accumulated by each invoking.

2. There is a value passed to the default argument
The name "d" will be bound to the passed object, it's proven from the
different identity showd above.

I think we could eliminate such accumulation effect by changing the function
as follow:[color=blue][color=green][color=darkred]
>>> def f(d=[]):[/color][/color][/color]
d = d+[0]
print d

J.R.

[Stian Søiland]

Default parameters

* J.R. spake thusly:[color=blue][color=green]
> > def f(d=[]):
> > d.append(0)
> > print d
> > f()
> > f()
> > Explain results. When is d bound?[/color][/color]

When is this issue going to be resolved? Enough newbie-pythoners have
made this mistake now.

Why not evaluate the parameter lists at calltime instead of definition
time? This should work the same way as lambdas.
[color=blue][color=green][color=darkred]
>>> f = lambda: []
>>> a=f()
>>> b=f()
>>> a.append(1)
>>> print a, b[/color][/color][/color]
[1] []


Maybe this could be defined in a similar way to remind of the
"lazy-evaluation":

def getvalue(cls):
return "OK"

class SomeClass:
def blapp(something : getvalue(), other: []):
print something, other

This way, the lambda forms defined after 'something' and 'other' are
evaluated each time the function is called without supplying those
parameters.

The lambda forms could be evaluated as if within the class-block, and
therefore they might actually use other values defined within that
namespace. However, this might again confuse users, as subclassed
attributes and instance attributes would not be resolved that way.

(Note that default-parameters-lambdas by today does NOT resolve this way:
[color=blue][color=green][color=darkred]
>>> class Blapp:[/color][/color][/color]
.... ting = 15
.... def fix(self, per=lambda: ting):
.... print per()
....[color=blue][color=green][color=darkred]
>>> a = Blapp()
>>> a.fix()[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 4, in fix
File "<stdin>", line 3, in <lambda>
NameError: global name 'ting' is not defined



Finally, things could also be done like this, to avoid confusion to all
those new folks (our main goal):

def blapp(something =getvalue(), other=[]):

getvalue() should be called if something-paramer is not specified (ie.
expression something=getva lue() is evaluated), and likewise for other.


Although this would break existing code and need to be delayed to at
least 3.0 and implemented in the __future__.

I must say I can't see the reason to not delay evalution now that we
have nested scopes. This way, even this would work:


class A:
default_height= 100
default_width=2 00
def make_picture(se lf, height=default_ height,
width=default_w idth):
self.gui.do_bla bla(height, width)
set_default_wid th = classmethod(set _default_width)

a = A()
a.make_picture( )
A.default_width = 150
a.make_picture( ) # Now with new default width


One might argue that this could could benefit from resolving
self.default_wi dth instead, that would still require setting
height=None and testing inside make_picture.

--
Stian Søiland Being able to break security doesn't make
Trondheim, Norway you a hacker more than being able to hotwire
http://stain.portveien.to/ cars makes you an automotive engineer. [ESR]

[Terry Reedy]

Re: Default parameters


"Stian Søiland" <stain@stud.ntn u.no> wrote in
message
news:slrnbtvte6 .tnk.stain@ozel ot.stud.ntnu.no ...[color=blue]
> When is this issue going to be resolved? Enough[/color]
newbie-pythoners have[color=blue]
> made this mistake now.[/color]

I am puzzled as to why. When I learned Python, I
read something to the effect that default value
expressions are evaluated at definition time. I
understood that the resulting objects were saved
for later use (parameter binding) when needed (as
default for value not given). I believed this and
that was that.
[color=blue]
> Why not evaluate the parameter lists at calltime[/color]
instead of definition[color=blue]
> time?[/color]

Run time code belongs in the body of the function
and the corresponding byte code in the code
object. To me anyway.

Terry J. Reedy

[Greg Ewing (using news.cis.dfn.de)]

Re: Default parameters

Stian Søiland wrote:[color=blue]
> When is this issue going to be resolved? Enough newbie-pythoners have
> made this mistake now.[/color]

Changes are rarely if ever made to Python for the sole reason
of reducing newbie mistakes. There needs to be a payoff for
long-term use of the language as well.

In this case, evaluating the default args at call time would
have a negative payoff, since it would slow down every call to
the function in cases where the default value doesn't need
to be evaluated more than once.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand

http://www.cosc.canterbury.ac.nz/~greg

[Paul Rubin]

Re: Default parameters

"Greg Ewing (using news.cis.dfn.de )" <g2h5dqi002@sne akemail.com> writes:[color=blue]
> Changes are rarely if ever made to Python for the sole reason
> of reducing newbie mistakes.[/color]

print 3/2

[Paul Rubin]

Re: Default parameters

"Greg Ewing (using news.cis.dfn.de )" <g2h5dqi002@sne akemail.com> writes:[color=blue]
> In this case, evaluating the default args at call time would
> have a negative payoff, since it would slow down every call to
> the function in cases where the default value doesn't need
> to be evaluated more than once.[/color]

In those cases the compiler should notice it and generate appropriate
code to evaluate the default arg just once. In many of the cases it
can put a static value into the .pyc file.

[Bengt Richter]

Re: Default parameters

On Wed, 17 Dec 2003 06:20:54 +0000 (UTC), stain@stud.ntnu .no (Stian =?iso-8859-1?Q?S=F8iland?= ) wrote:
[color=blue]
>* J.R. spake thusly:[color=green][color=darkred]
>> > def f(d=[]):
>> > d.append(0)
>> > print d
>> > f()
>> > f()
>> > Explain results. When is d bound?[/color][/color]
>
>When is this issue going to be resolved? Enough newbie-pythoners have
>made this mistake now.[/color]
It works as designed. The default parameter value bindings are made a def time.
If you want to do them at call time, the idiom is

def f(d=None):
if d is None: d = []
d.append(0)
print d
[color=blue]
>
>Why not evaluate the parameter lists at calltime instead of definition
>time? This should work the same way as lambdas.[/color]
Lambdas do work the same as defs, except for the automatic name binding
and the limitation to an expression as the body.[color=blue]
>[color=green][color=darkred]
>>>> f = lambda: []
>>>> a=f()
>>>> b=f()
>>>> a.append(1)
>>>> print a, b[/color][/color]
>[1] [][/color]
The comparable def code to your lambda would be
def f(): return []
The above is misguiding attention away from your point, IMO.
[color=blue]
>Maybe this could be defined in a similar way to remind of the
>"lazy-evaluation":
>
>def getvalue(cls):
> return "OK"
>
>class SomeClass:
> def blapp(something : getvalue(), other: []):
> print something, other
>[/color]
This might be an interesting concise spelling to accomplish what the following does:
(BTW, you need a self for normal methods)
[color=blue][color=green][color=darkred]
>>> class Defer(object):[/color][/color][/color]
... def __init__(self, fun): self.fun = fun
...[color=blue][color=green][color=darkred]
>>> class SomeClass(objec t):[/color][/color][/color]
... def blapp(self, something=Defer (lambda: getvalue()), other=Defer(lam bda:[])):
... if isinstance(some thing, Defer): something = something.fun()
... if isinstance(othe r, Defer): other = other.fun()
... print something, other
...[color=blue][color=green][color=darkred]
>>> def getvalue(): return 'gotten_value'[/color][/color][/color]
...[color=blue][color=green][color=darkred]
>>> sc = SomeClass()
>>> sc.blapp()[/color][/color][/color]
gotten_value [][color=blue][color=green][color=darkred]
>>> sc.blapp(1)[/color][/color][/color]
1 [][color=blue][color=green][color=darkred]
>>> sc.blapp('one', 'two')[/color][/color][/color]
one two
[color=blue]
>This way, the lambda forms defined after 'something' and 'other' are
>evaluated each time the function is called without supplying those
>parameters.
>
>The lambda forms could be evaluated as if within the class-block, and
>therefore they might actually use other values defined within that
>namespace. However, this might again confuse users, as subclassed
>attributes and instance attributes would not be resolved that way.[/color]
Yes, messy. ISTM better to let them be defined in the same scope
as the def, as in my example above.
[color=blue]
>
>(Note that default-parameters-lambdas by today does NOT resolve this way:
>[color=green][color=darkred]
>>>> class Blapp:[/color][/color]
>... ting = 15
>... def fix(self, per=lambda: ting):
>... print per()
>...[color=green][color=darkred]
>>>> a = Blapp()
>>>> a.fix()[/color][/color]
>Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "<stdin>", line 4, in fix
> File "<stdin>", line 3, in <lambda>
> NameError: global name 'ting' is not defined[/color]
Right, though you could get the functionality if you wanted to.[color=blue]
>
>
>
>Finally, things could also be done like this, to avoid confusion to all
>those new folks (our main goal):[/color]
If they're confused because their preconceptions are filtering out
anything discordant, pandering to them would not serve the language.
[color=blue]
>
>def blapp(something =getvalue(), other=[]):
>
>getvalue() should be called if something-paramer is not specified (ie.
>expression something=getva lue() is evaluated), and likewise for other.[/color]
No. I like your version with the lambda-colons much better.[color=blue]
>
>
>Although this would break existing code and need to be delayed to at
>least 3.0 and implemented in the __future__.
>
>I must say I can't see the reason to not delay evalution now that we[/color]
Are you sure you counted your negatives there? ;-)
[color=blue]
>have nested scopes. This way, even this would work:
>[/color]
You're assuming default_height will refer to A.default_heigh t[color=blue]
>
>class A:
> default_height= 100
> default_width=2 00
> def make_picture(se lf, height=default_ height,
> width=default_w idth):
> self.gui.do_bla bla(height, width)
> set_default_wid th = classmethod(set _default_width)
>[/color]
IMO better:

class A:
default_height= 100
default_width=2 00
def make_picture(se lf, height: A.default_heigh t,
width: A.default_width ):
self.gui.do_bla bla(height, width)
set_default_wid th = classmethod(set _default_width)
[color=blue]
>a = A()
>a.make_picture ()
>A.default_widt h = 150
>a.make_picture () # Now with new default width
>
>
>One might argue that this could could benefit from resolving
>self.default_w idth instead, that would still require setting
>height=None and testing inside make_picture.[/color]

Or maybe have the implict lambda have an implicit self arg bound like a method
if it is the default arg of a method, ie., so you could write

class A:
default_height= 100
default_width=2 00
def make_picture(se lf, height: self.default_he ight,
width: self.default_wi dth):
self.gui.do_bla bla(height, width)

Then when the function was called, it would be like getting an implicit something like
[color=blue][color=green][color=darkred]
>>> class A(object):[/color][/color][/color]
... default_height= 100
... default_width=2 00
... def make_picture(se lf, height=Defer(la mbda self: self.default_he ight),
... width=Defer(lam bda self: self.default_wi dth)):
... if isinstance(heig ht, Defer): height = height.fun.__ge t__(self)()
... if isinstance(widt h, Defer): width = width.fun.__get __(self)()
... self.gui.do_bla bla(height, width)
... class gui(object): # something to catch the above ;-/
... def do_blabla(h,w): print 'h=%r, w=%r'%(h,w)
... do_blabla = staticmethod(do _blabla)
...[color=blue][color=green][color=darkred]
>>> a=A()
>>> a.make_picture( )[/color][/color][/color]
h=100, w=200[color=blue][color=green][color=darkred]
>>> a.make_picture( 'one')[/color][/color][/color]
h='one', w=200[color=blue][color=green][color=darkred]
>>> a.make_picture( 'one','two')[/color][/color][/color]
h='one', w='two'

Obviously the nested class gui is just to make the self.gui.do_bla bla call work as spelled ;-)

I doubt if this is going to make it, but I think it's feasible without backwards breakage.
Write a PEP if you want to pursue something like that, but don't get overexcited ;-)

Regards,
Bengt Richter

[Alan Gauld]

Re: Beginner question - How to effectively pass a large list

On Tue, 16 Dec 2003 16:21:00 +0800, "J.R." <j.r.gao@motoro la.com>
wrote:[color=blue]
> Actually, the python is passing the identity (i.e. memory address) of each
> parameter, and it will bind to a local name within the function.
>
> Right?[/color]

Nope.
This is one case where understanding something of the insides of
Python helps. Basically Python variables are dictionary entries.
The variable values are the the dictionary values associated with
the variable names which are the dictionary keys.

Thus when you pass an argument to a function you are passing a
dictionary key. When the function uses the argument it looks up
the dictionary and uses the value found there.

This applies to all sorts of things in Python including modules -
a local dictionary associated with the module, and classes -
another dictionary. Dictionaries are fundamental to how Python
works and memory addresses per se play no part in the procedings.

HTH

Alan G.

Author of the Learn to Program website

http://www.freenetpages.co.uk/hp/alan.gauld