Estimating memory use?

[Roy Smith]

I've got a large text processing task to attack (it's actually a genomics
task; matching DNA probes against bacterial genomes). I've got roughly
200,000 probes, each of which is a 25 character long text string. My first
thought is to compile these into 200,000 regexes, but before I launch into
that, I want to do a back of the envelope guess as to how much memory that
will take.

Is there any easy way to find out how much memory a Python object takes?
If there was, it would be simple to compile a random small collection of
these patterns (say, 100 of them), and add up the sizes of the resulting
regex objects to get a rough idea of how much memory I'll need. I realize
I could just compile them all and watch the size of the Python process
grow, but that seems somewhat brute force.

[Tim N. van der Leeuw]

Re: Estimating memory use?

Hi,

What is your 'static' data (database), and what is your input-data?
Those 200.000 probes are your database? Perhaps they can be stored as
pickled compiled regexes and thus be loaded in pickled form; then you
don't need to keep them all in memory at once -- if you fear that
memory usage will be too big.

I don't know if perhaps other string-matching techniques can be used
btw; you don't need the full power of regexes I guess to match DNA
string patterns.
Perhaps you should investigate that a bit, and do some performance
tests?

cheers,

--Tim

[Alex Martelli]

Re: Estimating memory use?

Roy Smith <roy@panix.co m> wrote:
...[color=blue]
> Is there any easy way to find out how much memory a Python object takes?[/color]

No, but there are a few early attempts out there at supplying SOME ways
(not necessarily "easy", but SOME). For example, PySizer, at
<http://pysizer.8325.or g/>.


Alex

[Roy Smith]

Re: Estimating memory use?

In article <1h6olhu.12xxjw h12leq0gN%aleax @mail.comcast.n et>,
aleax@mail.comc ast.net (Alex Martelli) wrote:
[color=blue]
> Roy Smith <roy@panix.co m> wrote:
> ...[color=green]
> > Is there any easy way to find out how much memory a Python object takes?[/color]
>
> No, but there are a few early attempts out there at supplying SOME ways
> (not necessarily "easy", but SOME). For example, PySizer, at
> <http://pysizer.8325.or g/>.
>
>
> Alex
>[/color]

Looks interesting, thanks.

I've already discovered one (very) surprising thing -- if I build a dict
containing all my regexes (takes about 3 minutes on my PowerBook) and
pickle them to a file, re-loading the pickle takes just about as long as
compiling them did in the first place.

[Fredrik Lundh]

Re: Estimating memory use?

Roy Smith wrote:
[color=blue]
> I've already discovered one (very) surprising thing -- if I build a dict
> containing all my regexes (takes about 3 minutes on my PowerBook) and
> pickle them to a file, re-loading the pickle takes just about as long as
> compiling them did in the first place.[/color]

the internal RE byte code format is version dependent, so pickle stores the
patterns instead.

</F>

[MrJean1]

Re: Estimating memory use?

There is a function mx_sizeof() in the mx.Tools module from eGenix
which may be helpful. More at


<http://www.egenix.com/files/python/eGenix-mx-Extensions.html #mxTools>

/Jean Brouwers


PS) This is an approximation for memory usage which is useful in
certain, simple cases.

Each built-in type has an attribute __basicsize__ which is the size in
bytes needed to represent the basic type. For example
str.__basicsize __ returns 24 and int.__basictype __ returns 12.

However, __basicsize__ does not include the space needed to store the
object value. For a string, the length of the string has to be added
(times the character width). For example, the size of string "abcd"
would at least approximately str.__basicsize __ + len("abcd") bytes,
assuming single byte characters.

In addition, memory alignment should be taken into account by rounding
the size up to the next multiple of 8 (or maybe 16, depending on
platform, etc.).

An approximation for the amount of memory used by a string S (of single
byte characters) aligned to A bytes would be

(str.__basicsiz e__ + len(S) + A - 1) & A

Things are more complicated for types like list, tuple and dict and
instances of a class.

[MrJean1]

Re: Estimating memory use?

The name of the function in mx.Tools is sizeof() and not mx_sizeof().
My apologies.

Also, it turns out that the return value of mx.Tools.sizeof () function
is non-aligned. For example mx.Tools.sizeof ("abcde") returns 29 which
is fine, but not entirely "accurate".

/Jean Brouwers

[François Pinard]

Re: Estimating memory use?

[Fredrik Lundh]
[color=blue]
> the internal RE byte code format is version dependent, so pickle
> stores the patterns instead.[/color]

Oh! Nice to know. That explains why, when I was learning Python, my
initial experiment with pickles left me with the (probably wrong)
feeling that they were not worth the trouble.

It might be worth a note in the documentation, somewhere appropriate.

--
François Pinard http://pinard.progiciels-bpi.ca