Size limit on compiling?

[Leif K-Brooks]

I have a Python file of 1.2MB (it's a static database, not code, so it
really does have to be that big). It doesn't seem to get compiled when
imported (there's no .pyc file created), so importing it takes a few
seconds which I would really like to avoid. Is Python not compiling it
because of the large size? Is it because the whole file is on one really
long line? How can I fix this?

[Aahz]

Re: Size limit on compiling?

In article <Ijb%b.75$a6.33 911@newshog.new sread.com>,
Leif K-Brooks <eurleif@ecritt ers.biz> wrote:[color=blue]
>
>I have a Python file of 1.2MB (it's a static database, not code, so it
>really does have to be that big). It doesn't seem to get compiled when
>imported (there's no .pyc file created), so importing it takes a few
>seconds which I would really like to avoid. Is Python not compiling it
>because of the large size? Is it because the whole file is on one really
>long line? How can I fix this?[/color]

Use a real database, or at least pickle.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

"Do not taunt happy fun for loops. Do not change lists you are looping over."
--Remco Gerlich, comp.lang.pytho n

[Peter Hansen]

Re: Size limit on compiling?

Leif K-Brooks wrote:[color=blue]
>
> I have a Python file of 1.2MB (it's a static database, not code, so it
> really does have to be that big). It doesn't seem to get compiled when
> imported (there's no .pyc file created), so importing it takes a few
> seconds which I would really like to avoid. Is Python not compiling it
> because of the large size? Is it because the whole file is on one really
> long line? How can I fix this?[/color]

If it's the main file, it won't get compiled anyway. Only files
that are imported are compiled. In any case, see Aahz' answer for the
best approach.

-Peter

[Bengt Richter]

Re: Size limit on compiling?

On Thu, 26 Feb 2004 09:31:02 -0500, Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
>Leif K-Brooks wrote:[color=green]
>>
>> I have a Python file of 1.2MB (it's a static database, not code, so it
>> really does have to be that big). It doesn't seem to get compiled when
>> imported (there's no .pyc file created), so importing it takes a few
>> seconds which I would really like to avoid. Is Python not compiling it
>> because of the large size? Is it because the whole file is on one really
>> long line? How can I fix this?[/color]
>
>If it's the main file, it won't get compiled anyway. Only files
>that are imported are compiled. In any case, see Aahz' answer for the
>best approach.
>[/color]
If your file is highly structured (or you can separately prepare a version that is),
you might consider accessing it via mmap and struct (or even without struct,
if it's e.g., a flat array of fixed-length strings). You could enhance this
by pre-computing an index and storing it at the end or beginning of the file,
maybe data for a dict of (offset,length) values, similarly packed/retrieved
from the raw binary file (byte string) format. If you open the file read-only,
an efficient OS will not even page in or allocate swap file space for anything
you don't access (though 1.2 mb these days is not a big worry ;-).
But if your data is complicated, pretty soon you'd be reinventing pickle
and various db things, and you might as well use what's available already.

Regards,
Bengt Richter