Python object overhead?

[Bruno Desthuilliers]

Re: Python object overhead?

Bruno Desthuilliers a écrit :

Matt Garman a écrit :

(snip)

class FileRecord(obje ct):
>

> def __init__(self, line):
> self.line = line

>
If this is your real code, I don't see any reason why this should eat up
3 times more space than the original version.
>

Hem... Forget about this comment - not enough coffein yet I'm afraid.

[Matt Garman]

Re: Python object overhead?

On 3/23/07, Bjoern Schliessmann
<usenet-mail-0306.20.chr0n0s s@spamgourmet.c omwrote:

"one blank line" == "EOF"? That's strange. Intended?

In my case, I know my input data doesn't have any blank lines.
However, I'm glad you (and others) clarified the issue, because I
wasn't aware of the better methods for checking for EOF.

Example 2: read lines into objects:
# begin readobjects.py
import sys, time
class FileRecord:
def __init__(self, line):
self.line = line

>
What's this class intended to do?

Store a line :) I just wanted to post two runnable examples. So the
above class's real intention is just to be a (contrived) example.

In the program I actually wrote, my class structure was a bit more
interesting. After storing the input line, I'd then call split("|")
(to tokenize the line). Each token would then be assigned to an
member variable. Some of the member variables turned into ints or
floats as well.

My input data had three record types; all had a few common attributes.
So I created a parent class and three child classes.

Also, many folks have suggested operating on only one line at a time
(i.e. not storing the whole data set). Unfortunately, I'm constantly
"looking" forward and backward in the record set while I process the
data (i.e., to process any particular record, I sometimes need to know
the whole contents of the file). (This is purchased proprietary
vendor data that needs to be converted into our own internal format.)

Finally, for what it's worth: the total run time memory requirements
of my program is roughly 20x the datafile size. A 200MB file
literally requires 4GB of RAM to effectively process. Note that, in
addition to the class structure I defined above, I also create two
caches of all the data (two dicts with different keys from the
collection of objects). This is necessary to ensure the program runs
in a semi-reasonable amount of time.

Thanks to all for your input and suggestions. I received many more
responses than I expected!

Matt

[Bruno Desthuilliers]

Re: Python object overhead?

Matt Garman a écrit :
(snip)

Also, many folks have suggested operating on only one line at a time
(i.e. not storing the whole data set). Unfortunately, I'm constantly
"looking" forward and backward in the record set while I process the
data (i.e., to process any particular record, I sometimes need to know
the whole contents of the file). (This is purchased proprietary
vendor data that needs to be converted into our own internal format.)

Don't know if this could solve your problem, but have considered using
an intermediate (preferably embedded) SQL database (something like
SQLite) ?

[Bjoern Schliessmann]

Re: Python object overhead?

Matt Garman wrote:

In my case, I know my input data doesn't have any blank lines.

8)

I work with a (not self-written) perl script that does funny things
with blank lines in input files. Yeah, blank lines "aren't supposed
to" be in the input data ...

However, I'm glad you (and others) clarified the issue, because I
wasn't aware of the better methods for checking for EOF.

The principle was okay (to check if the string is totally empty). I
always used readlines so far and didn't have the problem.

Thanks to all for your input and suggestions. I received many
more responses than I expected!

You're welcome. :)

Regards,


Björn

--
BOFH excuse #374:

It's the InterNIC's fault.