synching with os.walk()

[Andre Meyer]

Hi all

os.walk() is a nice generator for performing actions on all files in a
directory and subdirectories. However, how can one use os.walk() for walking
through two hierarchies at once? I want to synchronise two directories (just
backup for now), but cannot see how I can traverse a second one. I do this
now with os.listdir() recursively, which works fine, but I am afraid that
recursion can become inefficient for large hierarchies.

thanks for your help
André

[120psi@gmail.com]

Re: synching with os.walk()

os.walk() is a nice generator for performing actions on all files in a

directory and subdirectories. However, how can one use os.walk() for walking
through two hierarchies at once? I want to synchronise two directories (just
backup for now), but cannot see how I can traverse a second one. I do this
now with os.listdir() recursively, which works fine, but I am afraid that
recursion can become inefficient for large hierarchies.

I've run into wanting to work with parallel directory structures
before, and what I generally do is something like:

for root, dirs, files in os.walk( dir1 ):
dir2_root = dir2 + root[len(dir1):]
for f in files:
dir1_path = os.path.join( root, f )
dir2_path = os.path.join( dir2_root, f )

Does this work for your needs?
-- Nils

[Paddy]

Re: synching with os.walk()


Andre Meyer wrote:

Hi all
>
os.walk() is a nice generator for performing actions on all files in a
directory and subdirectories. However, how can one use os.walk() for walking
through two hierarchies at once? I want to synchronise two directories (just
backup for now), but cannot see how I can traverse a second one. I do this
now with os.listdir() recursively, which works fine, but I am afraid that
recursion can become inefficient for large hierarchies.
>
thanks for your help
André

Walk each tree individually gathering file names relative to the head
of the tree and modification data.

compare the two sets of data to generate:
1. A list of what needs to be copied from the original to the copy.
2. A list of what needs to be copied from the copy to the original

Do the copying.

|You might want to show the user what needs to be done and give them
the option of aborting after generating the copy lists.

- Paddy.

[Paddy]

Re: synching with os.walk()


Paddy wrote:

Andre Meyer wrote:
>

Hi all

os.walk() is a nice generator for performing actions on all files in a
directory and subdirectories. However, how can one use os.walk() for walking
through two hierarchies at once? I want to synchronise two directories (just
backup for now), but cannot see how I can traverse a second one. I do this
now with os.listdir() recursively, which works fine, but I am afraid that
recursion can become inefficient for large hierarchies.

thanks for your help
André

>
Walk each tree individually gathering file names relative to the head
of the tree and modification data.
>
compare the two sets of data to generate:
1. A list of what needs to be copied from the original to the copy.
2. A list of what needs to be copied from the copy to the original
>
Do the copying.
>
|You might want to show the user what needs to be done and give them
the option of aborting after generating the copy lists.
>
- Paddy.

P.S. If you are on a Unix type system you can use tar to do the copying
as you can easily compress the data if it needs to go over a sow link,
and tar will take care of creating any needed directories in the
destination if you create new directories as well as new files.
- Paddy.

[Thomas Ploch]

Re: synching with os.walk()

>os.walk() is a nice generator for performing actions on all files in a
>directory and subdirectories. However, how can one use os.walk() for walking
>through two hierarchies at once? I want to synchronise two directories (just
>backup for now), but cannot see how I can traverse a second one. I do this
>now with os.listdir() recursively, which works fine, but I am afraid that
>recursion can become inefficient for large hierarchies.

>
I've run into wanting to work with parallel directory structures
before, and what I generally do is something like:
>
for root, dirs, files in os.walk( dir1 ):
dir2_root = dir2 + root[len(dir1):]
for f in files:
dir1_path = os.path.join( root, f )
dir2_path = os.path.join( dir2_root, f )
>

Wouldn't it be better to implement tree traversing into a class, then
you can traverse two directory trees at once and can do funny things
with it?

Thomas

[Paddy]

Re: synching with os.walk()


Paddy wrote:

P.S. If you are on a Unix type system you can use tar to do the copying
as you can easily compress the data if it needs to go over a sow link,

Sow links, transfers your data and then may form a tasty sandwich when
cooked.

(The original should, of course, read ...slow...)
- Pad.

[Antoine De Groote]

Re: synching with os.walk()

Andre Meyer wrote:

Hi all
>
os.walk() is a nice generator for performing actions on all files in a
directory and subdirectories. However, how can one use os.walk() for
walking through two hierarchies at once? I want to synchronise two
directories (just backup for now), but cannot see how I can traverse a
second one. I do this now with os.listdir() recursively, which works
fine, but I am afraid that recursion can become inefficient for large
hierarchies.
>
thanks for your help
André
>

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/191017

might be what you are looking for, or at least a starting point...

Regards,
antoine

[BartlebyScrivener]

Re: synching with os.walk()

Antoine De Groote wrote:

>

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/191017

might be what you are looking for, or at least a starting point...
>

There's an updated version of this script at pages 403-04 of the Python
Cookbook 2nd Edition.

rd

[petercable@gmail.com]

Re: synching with os.walk()



On Nov 24, 7:57 am, "Andre Meyer" <m...@acm.orgwr ote:

>
os.walk() is a nice generator for performing actions on all files in a
directory and subdirectories. However, how can one use os.walk() for walking
through two hierarchies at once? I want to synchronise two directories (just
backup for now), but cannot see how I can traverse a second one. I do this
now with os.listdir() recursively, which works fine, but I am afraid that
recursion can become inefficient for large hierarchies.
>

I wrote a script to perform this function using the dircmp class in the
filecmp module. I did something similar to this:
import filecmp, os, shutil

def backup(d1,d2):
print 'backing up %s to %s' % (d1,d2)
compare = filecmp.dircmp( d1,d2)
for item in compare.left_on ly:
fullpath = os.path.join(d1 , item)
if os.path.isdir(f ullpath):
shutil.copytree (fullpath,os.pa th.join(d2,item ))
elif os.path.isfile( fullpath):
shutil.copy2(fu llpath,d2)
for item in compare.diff_fi les:
shutil.copy2(os .path.join(d1,i tem),d2)
for item in compare.common_ dirs:
backup(os.path. join(d1,item),o s.path.join(d2, item))

if __name__ == '__main__':
import sys
if len(sys.argv) == 3:
backup(sys.argv[1], sys.argv[2])

My script has some error checking and keeps up to 5 previous versions
of a changed file. I find it very efficient, even with recursion, as it
only actually copies those files that have changed. I sync somewhere
around 5 GB worth of files nightly across the network and I haven't had
any trouble.

Of course, if I just had rsync available, I would use that.

Hope this helps,

Pete