Sorting directory contents

[Wolfgang Draxinger]

H folks,

I got, hmm not really a problem, more a question of elegance:

In a current project I have to read in some files in a given
directory in chronological order, so that I can concatenate the
contents in those files into a new one (it's XML and I have to
concatenate some subelements, about 4 levels below the root
element). It all works, but somehow I got the feeling, that my
solution is not as elegant as it could be:

src_file_paths = dict()
for fname in os.listdir(sour cedir):
fpath = sourcedir+os.se p+fname
if not match_fname_pat tern(fname): continue
src_file_paths[os.stat(fpath). st_mtime] = fpath
for ftime in src_file_paths. keys().sort():
read_and_concat enate(src_file_ paths[ftime])

of course listdir and sorting could be done in a separate
function, but I wonder if there was a more elegant approach.

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867

[Jussi Salmela]

Re: Sorting directory contents

Wolfgang Draxinger kirjoitti:

H folks,
>
I got, hmm not really a problem, more a question of elegance:
>
In a current project I have to read in some files in a given
directory in chronological order, so that I can concatenate the
contents in those files into a new one (it's XML and I have to
concatenate some subelements, about 4 levels below the root
element). It all works, but somehow I got the feeling, that my
solution is not as elegant as it could be:
>
src_file_paths = dict()
for fname in os.listdir(sour cedir):
fpath = sourcedir+os.se p+fname
if not match_fname_pat tern(fname): continue
src_file_paths[os.stat(fpath). st_mtime] = fpath
for ftime in src_file_paths. keys().sort():
read_and_concat enate(src_file_ paths[ftime])
>
of course listdir and sorting could be done in a separate
function, but I wonder if there was a more elegant approach.
>
Wolfgang Draxinger

I'm not claiming the following to be more elegant, but I would do it
like this (not tested!):

src_file_paths = dict()
prefix = sourcedir + os.sep
for fname in os.listdir(sour cedir):
if match_fname_pat tern(fname):
fpath = prefix + fname
src_file_paths[os.stat(fpath). st_mtime] = fpath
for ftime in src_file_paths. keys().sort():
read_and_concat enate(src_file_ paths[ftime])


Cheers,
Jussi

[Wolfgang Draxinger]

Re: Sorting directory contents

Jussi Salmela wrote:

I'm not claiming the following to be more elegant, but I would
do it like this (not tested!):
>
src_file_paths = dict()
prefix = sourcedir + os.sep
for fname in os.listdir(sour cedir):
if match_fname_pat tern(fname):
fpath = prefix + fname
src_file_paths[os.stat(fpath). st_mtime] = fpath
for ftime in src_file_paths. keys().sort():
read_and_concat enate(src_file_ paths[ftime])

Well, both versions, mine and yours won't work as it was written
down, as they neglegt the fact, that different files can have
the same st_mtime and that <listtype>.sort () doesn't return a
sorted list.

However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically , then alphabetically.

def listdir_chrono( dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirp ath):
mtime = os.stat(dirpath +os.sep+fname). st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)

mtimes = files_dict.keys ()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.appen d(fname)
return filenames

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867

[Larry Bates]

Re: Sorting directory contents

Wolfgang Draxinger wrote:

Jussi Salmela wrote:
>

>I'm not claiming the following to be more elegant, but I would
>do it like this (not tested!):
>>
>src_file_pat hs = dict()
>prefix = sourcedir + os.sep
>for fname in os.listdir(sour cedir):
> if match_fname_pat tern(fname):
> fpath = prefix + fname
> src_file_paths[os.stat(fpath). st_mtime] = fpath
>for ftime in src_file_paths. keys().sort():
> read_and_concat enate(src_file_ paths[ftime])

>
Well, both versions, mine and yours won't work as it was written
down, as they neglegt the fact, that different files can have
the same st_mtime and that <listtype>.sort () doesn't return a
sorted list.
>
However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically , then alphabetically.
>
def listdir_chrono( dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirp ath):
mtime = os.stat(dirpath +os.sep+fname). st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)
>
mtimes = files_dict.keys ()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.appen d(fname)
return filenames
>
Wolfgang Draxinger

Four suggestions:

1) You might want to use os.path.join(di rpath, fname) instead of
dirpath+os.sep+ fname.

2) You may be able to use glob.glob(<patt ern>) to filter the files
more easily.

3) You didn't handle the possibility that there is s subdirectory
in the current directory. You need to check to make sure it is
a file you are processing as os.listdir() returns files AND
directories.

4) If you just put a tuple containing (mtime, filename) in a list
each time through the loop you can just sort that list at the
end it will be sorted by mtime and then alphabetically.

Example (not tested):

def listdir_chrono( dirpath):
import os
#
# Get a list of full pathnames for all the files in dirpath
# and exclude all the subdirectories. Note: This might be
# able to be replaced by glob.glob() to simplify. I would then
# add a second optional parameter: mask="" that would allow me
# to pass in a mask.
#
# List comprehensions are our friend when we are processing
# lists of things.
#
files=[os.path.join(di rpath, x) for x in os.listdir(dirp ath)
if not os.path.isdir(o s.path.join(dir path, x)]

#
# Get a list of tuples that contain (mtime, filename) that
# I can sort.
#
flist=[(os.stat(x).st_ mtime, x) for x in files]

#
# Sort them. Sort will sort on mtime, then on filename
#
flist.sort()
#
# Extract a list of the filenames only and return it
#
return [x[1] for x in flist]
#
# or if you only want the basenames of the files
#
#return [os.path.basenam e(x[1]) for x in flist]



-Larry Bates

[Paul Rubin]

Re: Sorting directory contents

Wolfgang Draxinger <wdraxinger@dar kstargames.dewr ites:

src_file_paths = dict()
for fname in os.listdir(sour cedir):
fpath = sourcedir+os.se p+fname
if not match_fname_pat tern(fname): continue
src_file_paths[os.stat(fpath). st_mtime] = fpath
for ftime in src_file_paths. keys().sort():
read_and_concat enate(src_file_ paths[ftime])

Note you have to used sorted() and not .sort() to get back a value
that you can iterate through.

Untested:

from itertools import ifilter

goodfiles = ifilter(match_f name_pattern,
(sourcedir+os.s ep+fname for \
fname in os.listdir(sour cedir))

for f,t in sorted((fname,o s.stat(f).st_mt ime) for fname in goodfiles,
key=lambda (fname,ftime): ftime):
read_and_concat enate(f)

If you're a lambda-phobe you can use operator.itemge tter(1) instead of
the lambda. Obviously you don't need the separate goodfiles variable
but things get a bit deeply nested without it.

[Jussi Salmela]

Re: Sorting directory contents

Wolfgang Draxinger kirjoitti:

Jussi Salmela wrote:
>

>I'm not claiming the following to be more elegant, but I would
>do it like this (not tested!):
>>
>src_file_pat hs = dict()
>prefix = sourcedir + os.sep
>for fname in os.listdir(sour cedir):
> if match_fname_pat tern(fname):
> fpath = prefix + fname
> src_file_paths[os.stat(fpath). st_mtime] = fpath
>for ftime in src_file_paths. keys().sort():
> read_and_concat enate(src_file_ paths[ftime])

>
Well, both versions, mine and yours won't work as it was written
down, as they neglegt the fact, that different files can have
the same st_mtime and that <listtype>.sort () doesn't return a
sorted list.
>
However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically , then alphabetically.
>
def listdir_chrono( dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirp ath):
mtime = os.stat(dirpath +os.sep+fname). st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)
>
mtimes = files_dict.keys ()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.appen d(fname)
return filenames
>
Wolfgang Draxinger

More elegant or not ... I did it MY WAYYYY!!! (and tested this time
really carefully;)):

#-------------------------------
def listdir_chrono_ 2(dirpath):
import os
files_dict = {}
prefix = dirpath + os.sep
for fname in os.listdir(dirp ath):
mtime = os.stat(prefix + fname).st_mtime
files_dict.setd efault(mtime, []).append(fname)

mtimes = sorted(files_di ct.keys())
filenames = []
for mtime in mtimes:
filenames += sorted(files_di ct[mtime])
return filenames

firstLst = listdir_chrono( '.')
secondLst = listdir_chrono_ 2('.')
if firstLst == secondLst: print 'OK'
else: print 'ERROR!!!'
#-------------------------------

I keep taking the "dirpath + os.sep" part out of the loop because it is
a loop invariant and doesn't have to be inside the loop.

Cheer

[Rob Wolfe]

Re: Sorting directory contents


Wolfgang Draxinger wrote:

However this code works (tested) and behaves just like listdir,
only that it sorts files chronologically , then alphabetically.
>
def listdir_chrono( dirpath):
import os
files_dict = dict()
for fname in os.listdir(dirp ath):
mtime = os.stat(dirpath +os.sep+fname). st_mtime
if not mtime in files_dict:
files_dict[mtime] = list()
files_dict[mtime].append(fname)
>
mtimes = files_dict.keys ()
mtimes.sort()
filenames = list()
for mtime in mtimes:
fnames = files_dict[mtime]
fnames.sort()
for fname in fnames:
filenames.appen d(fname)
return filenames

Using the builtin functions `sorted`, `filter` and the `setdefault`
method of dictionary could a little shorten your code:

def listdir_chrono( dirpath):
import os
files_dict = {}
for fname in filter(os.path. isfile, os.listdir(dirp ath)):
mtime = os.stat(os.path .join(dirpath, fname)).st_mtim e
files_dict.setd efault(mtime, []).append(fname)

filenames = []
for mtime in sorted(files_di ct):
for fname in sorted(files_di ct[mtime]):
filenames.appen d(fname)
return filenames

--
HTH,
Rob

[Wolfgang Draxinger]

Re: Sorting directory contents

Larry Bates wrote:

3) You didn't handle the possibility that there is s
subdirectory
in the current directory. You need to check to make sure it
is a file you are processing as os.listdir() returns files
AND directories.

Well, the directory the files are in is not supposed to have any
subdirectories.

4) If you just put a tuple containing (mtime, filename) in a
list
each time through the loop you can just sort that list at
the end it will be sorted by mtime and then alphabetically.

Ah, of course. Hmm, seems I was short of caffeine when I hacked
my code :-P

def listdir_chrono( dirpath):
import os
#
# Get a list of full pathnames for all the files in dirpath
# and exclude all the subdirectories. Note: This might be
# able to be replaced by glob.glob() to simplify. I would
# then add a second optional parameter: mask="" that would
# allow me to pass in a mask.
#
# List comprehensions are our friend when we are processing
# lists of things.
#
files=[os.path.join(di rpath, x) for x in
os.listdir(dirp ath)
if not os.path.isdir(o s.path.join(dir path, x)]
>
#
# Get a list of tuples that contain (mtime, filename) that
# I can sort.
#
flist=[(os.stat(x).st_ mtime, x) for x in files]
>
#
# Sort them. Sort will sort on mtime, then on filename
#
flist.sort()
#
# Extract a list of the filenames only and return it
#
return [x[1] for x in flist]
#
# or if you only want the basenames of the files
#
#return [os.path.basenam e(x[1]) for x in flist]

Now, THAT is elegant.

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867

[Peter Otten]

Re: Sorting directory contents

Wolfgang Draxinger wrote:

I got, hmm not really a problem, more a question of elegance:
>
In a current project I have to read in some files in a given
directory in chronological order, so that I can concatenate the
contents in those files into a new one (it's XML and I have to
concatenate some subelements, about 4 levels below the root
element). It all works, but somehow I got the feeling, that my
solution is not as elegant as it could be:
>
src_file_paths = dict()
for fname in os.listdir(sour cedir):
fpath = sourcedir+os.se p+fname
if not match_fname_pat tern(fname): continue
src_file_paths[os.stat(fpath). st_mtime] = fpath
for ftime in src_file_paths. keys().sort():
read_and_concat enate(src_file_ paths[ftime])
>
of course listdir and sorting could be done in a separate
function, but I wonder if there was a more elegant approach.

If glob.glob() is good enough to replace your custom match_fname_pat tern()
you can save a few steps:

pattern = os.path.join(so urcedir, "*.xml")
files = glob.glob(patte rn)
files.sort(key= os.path.getmtim e)
for fn in files:
read_and_concat enate(fn)

Peter

[Jussi Salmela]

Re: Sorting directory contents

Larry Bates kirjoitti:

Wolfgang Draxinger wrote:

>Jussi Salmela wrote:
>>

>>I'm not claiming the following to be more elegant, but I would
>>do it like this (not tested!):
>>>
>>src_file_path s = dict()
>>prefix = sourcedir + os.sep
>>for fname in os.listdir(sour cedir):
>> if match_fname_pat tern(fname):
>> fpath = prefix + fname
>> src_file_paths[os.stat(fpath). st_mtime] = fpath
>>for ftime in src_file_paths. keys().sort():
>> read_and_concat enate(src_file_ paths[ftime])

>Well, both versions, mine and yours won't work as it was written
>down, as they neglegt the fact, that different files can have
>the same st_mtime and that <listtype>.sort () doesn't return a
>sorted list.
>>
>However this code works (tested) and behaves just like listdir,
>only that it sorts files chronologically , then alphabetically.
>>
>def listdir_chrono( dirpath):
> import os
> files_dict = dict()
> for fname in os.listdir(dirp ath):
> mtime = os.stat(dirpath +os.sep+fname). st_mtime
> if not mtime in files_dict:
> files_dict[mtime] = list()
> files_dict[mtime].append(fname)
>>
> mtimes = files_dict.keys ()
> mtimes.sort()
> filenames = list()
> for mtime in mtimes:
> fnames = files_dict[mtime]
> fnames.sort()
> for fname in fnames:
> filenames.appen d(fname)
> return filenames
>>
>Wolfgang Draxinger

>
Four suggestions:
>
1) You might want to use os.path.join(di rpath, fname) instead of
dirpath+os.sep+ fname.
>
2) You may be able to use glob.glob(<patt ern>) to filter the files
more easily.
>
3) You didn't handle the possibility that there is s subdirectory
in the current directory. You need to check to make sure it is
a file you are processing as os.listdir() returns files AND
directories.
>
4) If you just put a tuple containing (mtime, filename) in a list
each time through the loop you can just sort that list at the
end it will be sorted by mtime and then alphabetically.
>
Example (not tested):
>
def listdir_chrono( dirpath):
import os
#
# Get a list of full pathnames for all the files in dirpath
# and exclude all the subdirectories. Note: This might be
# able to be replaced by glob.glob() to simplify. I would then
# add a second optional parameter: mask="" that would allow me
# to pass in a mask.
#
# List comprehensions are our friend when we are processing
# lists of things.
#
files=[os.path.join(di rpath, x) for x in os.listdir(dirp ath)
if not os.path.isdir(o s.path.join(dir path, x)]
>
#
# Get a list of tuples that contain (mtime, filename) that
# I can sort.
#
flist=[(os.stat(x).st_ mtime, x) for x in files]
>
#
# Sort them. Sort will sort on mtime, then on filename
#
flist.sort()
#
# Extract a list of the filenames only and return it
#
return [x[1] for x in flist]
#
# or if you only want the basenames of the files
#
#return [os.path.basenam e(x[1]) for x in flist]
>
>
>
-Larry Bates
>

And as in Peter Ottens glob.glob variation, this shortens considerably
by using sort with key instead of a separate list flist:

files.sort(key= lambda x:(os.stat(x).s t_mtime, x))

Cheers,
Jussi