randomly write to a file

[rohit]

hi,
i am developing a desktop search.For the index of the files i have
developed an algorithm with which
i should be able to read and write to a line if i know its line
number.
i can read a specified line by using the module linecache
but i am struck as to how to implement writing to the n(th) line in a
file EFFICIENTLY
which means i don't want to traverse the file sequentially to reach
the n(th) line

Please help.
Regards
Rohit

[kyosohma@gmail.com]

Re: randomly write to a file

On May 7, 2:51 pm, rohit <rohitsethi...@ gmail.comwrote:

hi,
i am developing a desktop search.For the index of the files i have
developed an algorithm with which
i should be able to read and write to a line if i know its line
number.
i can read a specified line by using the module linecache
but i am struck as to how to implement writing to the n(th) line in a
file EFFICIENTLY
which means i don't want to traverse the file sequentially to reach
the n(th) line
>
Please help.
Regards
Rohit

Hi,

Looking through the archives, it looks like some recommend reading the
file into a list and doing it that way. And if they file is too big,
than use a database. See links below:

http://mail.python.org/pipermail/tutor/2006-March/045571.html

http://mail.python.org/pipermail/tutor/2006-March/045572.html

I also found this interesting idea that explains what would be needed
to accomplish this task:

http://mail.python.org/pipermail/python-list/2001-April/076890.html

Have fun!

Mike

[Gabriel Genellina]

Re: randomly write to a file

En Mon, 07 May 2007 16:51:37 -0300, rohit <rohitsethidce@ gmail.com>
escribió:

i am developing a desktop search.For the index of the files i have
developed an algorithm with which
i should be able to read and write to a line if i know its line
number.
i can read a specified line by using the module linecache
but i am struck as to how to implement writing to the n(th) line in a
file EFFICIENTLY
which means i don't want to traverse the file sequentially to reach
the n(th) line

You can only replace a line in-place with another of exactly the same
length. If the lengths differ, you have to write the modified line and all
the following ones.
If all your lines are of fixed length, you have a "record". To read record
N (counting from 0):
a_file.seek(N*r ecord_length)
return a_file.read(rec ord_length)
And then you are reinventing ISAM.

--
Gabriel Genellina

[Nick Vatamaniuc]

Re: randomly write to a file

Rohit,

Consider using an SQLite database. It comes with Python 2.5 and
higher. SQLite will do a nice job keeping track of the index. You can
easily find the line you need with a SQL query and your can write to
it as well. When you have a file and you write to one line of the
file, all of the rest of the lines will have to be shifted to
accommodate, the potentially larger new line.

-Nick Vatamaniuc


On May 7, 3:51 pm, rohit <rohitsethi...@ gmail.comwrote:

hi,
i am developing a desktop search.For the index of the files i have
developed an algorithm with which
i should be able to read and write to a line if i know its line
number.
i can read a specified line by using the module linecache
but i am struck as to how to implement writing to the n(th) line in a
file EFFICIENTLY
which means i don't want to traverse the file sequentially to reach
the n(th) line
>
Please help.
Regards
Rohit

[rohit]

Re: randomly write to a file

nick,
i just wanted to ask for time constrained applications like searching
won't sqlite be a expensive approach.
i mean searching and editing o the files is less expensive by the time
taken .
so i need an approach which will allow me writing randomly to a line
in file without using a database
On May 8, 2:41 am, Nick Vatamaniuc <vatam...@gmail .comwrote:

Rohit,
>
Consider using an SQLite database. It comes with Python 2.5 and
higher. SQLite will do a nice job keeping track of the index. You can
easily find the line you need with a SQL query and your can write to
it as well. When you have a file and you write to one line of the
file, all of the rest of the lines will have to be shifted to
accommodate, the potentially larger new line.
>
-Nick Vatamaniuc
>

[rohit]

Re: randomly write to a file

hi gabriel,
i am utilizing file names and their paths which are written to a file
on a singe line.
now if i use records that would be wasting too much space as there is
no limit on the no. of characters (at max) in the path.
next best approach i can think of is reading the file in memory
editing it and writing the portion that has just been altered and the
followiing lines
but is there a better approach you can highlight?

You can only replace a line in-place with another of exactly the same
length. If the lengths differ, you have to write the modified line and all
the following ones.
If all your lines are of fixed length, you have a "record". To read record
N (counting from 0):
a_file.seek(N*r ecord_length)
return a_file.read(rec ord_length)
And then you are reinventing ISAM.
>
--
Gabriel Genellina

[Steven D'Aprano]

Re: randomly write to a file

On Mon, 07 May 2007 12:51:37 -0700, rohit wrote:

i can read a specified line by using the module linecache but i am
struck as to how to implement writing to the n(th) line in a file
EFFICIENTLY
which means i don't want to traverse the file sequentially to reach the
n(th) line

Unless you are lucky enough to be using an OS that supports random-access
line access to text files natively, if such a thing even exists, you
can't because you don't know how long each line will be.

If you can guarantee fixed-length lines, then you can use file.seek() to
jump to the appropriate byte position.

If the lines are random lengths, but you can control access to the files
so other applications can't write to them, you can keep an index table,
which you update as needed.

Otherwise, if the files are small enough, say up to 20 or 40MB each, just
read them entirely into memory.

Otherwise, you're out of luck.


--
Steven.

[Steven D'Aprano]

Re: randomly write to a file

On Mon, 07 May 2007 14:41:02 -0700, Nick Vatamaniuc wrote:

Rohit,
>
Consider using an SQLite database. It comes with Python 2.5 and higher.
SQLite will do a nice job keeping track of the index. You can easily
find the line you need with a SQL query and your can write to it as
well. When you have a file and you write to one line of the file, all of
the rest of the lines will have to be shifted to accommodate, the
potentially larger new line.

Using an database for tracking line number and byte position -- isn't
that a bit overkill?

I would have thought something as simple as a list of line lengths would
do:

offsets = [35, # first line is 35 bytes long
19, # second line is 19 bytes long...
45, 12, 108, 67]


To get to the nth line, you have to seek to byte position:

sum(offsets[:n])



--
Steven.

[Alex Martelli]

Re: randomly write to a file

Steven D'Aprano <steven@REMOVE. THIS.cybersourc e.com.auwrote:

On Mon, 07 May 2007 14:41:02 -0700, Nick Vatamaniuc wrote:
>

Rohit,

Consider using an SQLite database. It comes with Python 2.5 and higher.
SQLite will do a nice job keeping track of the index. You can easily
find the line you need with a SQL query and your can write to it as
well. When you have a file and you write to one line of the file, all of
the rest of the lines will have to be shifted to accommodate, the
potentially larger new line.

>
>
Using an database for tracking line number and byte position -- isn't
that a bit overkill?
>
I would have thought something as simple as a list of line lengths would
do:
>
offsets = [35, # first line is 35 bytes long
19, # second line is 19 bytes long...
45, 12, 108, 67]
>
>
To get to the nth line, you have to seek to byte position:
>
sum(offsets[:n])

....and then you STILL can't write there (without reading and rewriting
all the succeeding part of the file) unless the line you're writing is
always the same length as the one you're overwriting, which doesn't seem
to be part of the constraints in the OP's original application. I'm
with Nick in recommending SQlite for the purpose -- it _IS_ quite
"lite", as its name suggests. BSD-DB (a DB that's much more complicated
to use, being far lower-level, but by the same token affords you
extremely fine-grained control of operations) might be an alternative
IF, after first having coded the application with SQLite, you can indeed
prove, profiler in hand, that it's a serious bottleneck. However,
premature optimization is the root of all evil in programming.


Alex

[Steven D'Aprano]

Re: randomly write to a file

On Mon, 07 May 2007 20:00:57 -0700, Alex Martelli wrote:

Steven D'Aprano <steven@REMOVE. THIS.cybersourc e.com.auwrote:
>

>On Mon, 07 May 2007 14:41:02 -0700, Nick Vatamaniuc wrote:
>>

Rohit,
>
Consider using an SQLite database. It comes with Python 2.5 and
higher. SQLite will do a nice job keeping track of the index. You can
easily find the line you need with a SQL query and your can write to
it as well. When you have a file and you write to one line of the
file, all of the rest of the lines will have to be shifted to
accommodate, the potentially larger new line.

>>
>>
>Using an database for tracking line number and byte position -- isn't
>that a bit overkill?
>>
>I would have thought something as simple as a list of line lengths
>would do:
>>
>offsets = [35, # first line is 35 bytes long
> 19, # second line is 19 bytes long... 45, 12, 108, 67]
>>
>>
>To get to the nth line, you have to seek to byte position:
>>
>sum(offsets[:n])

>
...and then you STILL can't write there (without reading and rewriting
all the succeeding part of the file) unless the line you're writing is
always the same length as the one you're overwriting, which doesn't seem
to be part of the constraints in the OP's original application. I'm
with Nick in recommending SQlite for the purpose -- it _IS_ quite
"lite", as its name suggests.

Hang on, as I understand it, Nick just suggesting using SQlite for
holding indexes into the file! That's why I said it was overkill. So
whether the indexes are in a list or a database, you've _still_ got to
deal with writing to the file.

If I've misunderstood Nick's suggestion, if he actually meant to read the
entire text file into the database, well, that's just a heavier version
of reading the file into a list of strings, isn't it? If the database
gives you more and/or better functionality than file.readlines( ), then I
have no problem with using the right tool for the job.


--
Steven.

[Alex Martelli]

Re: randomly write to a file

Steven D'Aprano <steven@REMOVE. THIS.cybersourc e.com.auwrote:
...

Hang on, as I understand it, Nick just suggesting using SQlite for
holding indexes into the file! That's why I said it was overkill. So
whether the indexes are in a list or a database, you've _still_ got to
deal with writing to the file.
>
If I've misunderstood Nick's suggestion, if he actually meant to read the
entire text file into the database, well, that's just a heavier version
of reading the file into a list of strings, isn't it? If the database
gives you more and/or better functionality than file.readlines( ), then I
have no problem with using the right tool for the job.

Ah well, I may have misunderstood myself. I'd keep the whole thing in
an SQlite table, definitely NOT a table + an external file -- no, that's
not going to be heavier than reading things in memory, SQLite is smarter
than one might think:-). Obviously, I'm assuming that one's dealing
with an amount of data that doesn't just comfortably and easily fit in
memory, or at least one that gives pause at the thought of sucking it
all into memory and writing it back out again at every program run.


Alex