list2str and performance

[spr]

Hi,

I'm trying to learn Python and I'd appreciate any comments about my
small snippets of code. I read an old anecdote about performance here:

http://www.python.org/doc/essays/list2str/

First, I tried to figure out what would be the most pythonic approach by
today's standards:


def ListToStr(l):
"""Convert a list of integers into a string.
"""
return "".join([chr(n) for n in l])

def StrToList(s):
"""Convert a string into a list of integers.
"""
return [ord(c) for c in s]


By the way, using a generator expression in this case seem a bit slower
than a list comprehension and I'm not sure why.

I tried to improve the performance with Psyco and it became about 6
times as fast just for free. Then, I quickly tried Pyrex but I didn't
feel comfortable with using a dialect. So I trying Boost.Python, and as
I optimized the code it became more and more C-ish. The result is this:


// Convert a list of integers into a string.
PyObject* ListToStr(const boost::python:: list& l)
{
PyObject* s;
Py_BEGIN_ALLOW_ THREADS

const size_t length = PyList_GET_SIZE (l.ptr());

// Couldn't find a function for allocating a PyString from scratch.
s = (PyObject*)_PyO bject_NewVar(&P yString_Type, length);
((PyStringObjec t*)s)->ob_shash = -1;
((PyStringObjec t*)s)->ob_sstate = SSTATE_NOT_INTE RNED;
((PyStringObjec t*)s)->ob_sval[((PyStringObjec t*)s)->ob_size] = '\0';

char* s_items = PyString_AS_STR ING(s);
char* ps = s_items, *ps_end = ps + length;
PyIntObject** pl = (PyIntObject**) ((PyListObject* )l.ptr())->ob_item;

while (ps < ps_end)
{
*ps++ = (char)(*pl++)->ob_ival;
}

Py_END_ALLOW_TH READS
return s;
}

// Convert a string into a list of integers.
PyObject* StrToList(const boost::python:: str& s)
{
PyObject* l;
Py_BEGIN_ALLOW_ THREADS

const size_t length = PyString_GET_SI ZE(s.ptr());
l = PyList_New(leng th);

PyObject** pl = ((PyListObject* )l)->ob_item, **pl_end = pl + length;
unsigned char* ps = (unsigned char*)PyString_ AS_STRING(s.ptr ());

while (pl < pl_end)
{
*pl++ = PyInt_FromLong( *ps++);
}

Py_END_ALLOW_TH READS
return l;
}

Is it safe here to use Py_BEGIN_ALLOW_ THREADS and Py_END_ALLOW_TH READS?
On my machine this is about 50 times as fast as plain Python, but is
this as fast as it can get?

When performance matters and you have to develop a CPU-bound
application, do you think it is possible to eventually achieve nearly
the best performance by extending Python with C or C++ modules, or is it
better to take the embedding approach, that is, use a C or C++ core that
calls Python scripts?

[Fredrik Lundh]

Re: list2str and performance

"spr" wrote:

When performance matters and you have to develop a CPU-bound
application, do you think it is possible to eventually achieve nearly
the best performance by extending Python with C or C++ modules, or is it
better to take the embedding approach, that is, use a C or C++ core that
calls Python scripts?

the embedding approach only makes sense if you 1) have an existing application
that you want to extend with Python, or 2) are trying to sneak Python into an exist-
ing project (it's just a library, you know ;-).

for any other case, putting "Python at the top" is a lot more practical.

</F>