python noob, multiple file i/o

[hiro]

Hi there,

I'm very new to python, the problem I need to solve is whats the "best/
simplest/cleanest" way to read in multiple files (ascii), do stuff to
them, and write them out(ascii).

--
import os

filePath = ('O:/spam/eggs/')
for file in os.listdir(file Path): #straight from docs
# iterate the function through all the files in the directory
# write results to separate files <- this is where I'm mostly
stuck.

--
For clarity's sake, the file naming conventions for the files I'm
reading from are file.1.txt -file.nth.txt

It's been a long day, i'm at my wits end, so I apologize in advance if
I'm not making much sense here.
syntax would also be great if you can share some recipes.

Thanks in advance.

h.

[Jon Clements]

Re: python noob, multiple file i/o

On 16 Mar, 03:56, "hiro" <Nun...@gmail.c omwrote:

Hi there,
>
I'm very new to python, the problem I need to solve is whats the "best/
simplest/cleanest" way to read in multiple files (ascii), do stuff to
them, and write them out(ascii).
>
--
import os
>
filePath = ('O:/spam/eggs/')
for file in os.listdir(file Path): #straight from docs
# iterate the function through all the files in the directory
# write results to separate files <- this is where I'm mostly
stuck.
>
--
For clarity's sake, the file naming conventions for the files I'm
reading from are file.1.txt -file.nth.txt
>
It's been a long day, i'm at my wits end, so I apologize in advance if
I'm not making much sense here.
syntax would also be great if you can share some recipes.

I'd try the glob module.

import glob

# Get a list of filenames matching wildcard criteria
# (note that path is relative to working directory of program)
matching_file_list = glob.glob('O:/spam/eggs/*.txt')

# For each file that matches, open it and process it in some way...
for filename in matching_file_list:
infile = file(filename)
outfile = file(filename + '.out','w')
# Process the input file line by line...
for line in infile:
pass # Do something more useful here, change line and write to
outfile?
# Be explicit with file closures
outfile.close()
infile.close()

Of course, you can change the wild card criteria in the glob
statement, and also then filter further using regular expressions to
choose only files matching more specific criteria. This should be
enough to get you started though.

hth

Jon.

[Jon Clements]

Re: python noob, multiple file i/o

On 16 Mar, 09:02, "Jon Clements" <jon...@googlem ail.comwrote:

On 16 Mar, 03:56, "hiro" <Nun...@gmail.c omwrote:
>
>
>
>
>

Hi there,

>

I'm very new to python, the problem I need to solve is whats the "best/
simplest/cleanest" way to read in multiple files (ascii), do stuff to
them, and write them out(ascii).

>

--
import os

>

filePath = ('O:/spam/eggs/')
for file in os.listdir(file Path): #straight from docs
# iterate the function through all the files in the directory
# write results to separate files <- this is where I'm mostly
stuck.

>

--
For clarity's sake, the file naming conventions for the files I'm
reading from are file.1.txt -file.nth.txt

>

It's been a long day, i'm at my wits end, so I apologize in advance if
I'm not making much sense here.
syntax would also be great if you can share some recipes.

>
I'd try the glob module.
>

import glob
>
# Get a list of filenames matching wildcard criteria
# (note that path is relative to working directory of program)
matching_file_list = glob.glob('O:/spam/eggs/*.txt')
>
# For each file that matches, open it and process it in some way...
for filename in matching_file_list:
    infile = file(filename)
    outfile = file(filename + '.out','w')
    # Process the input file line by line...
    for line in infile:
        pass # Do something more useful here, change line and write to
outfile?
    # Be explicit with file closures
    outfile.close()
    infile.close()

>
Of course, you can change the wild card criteria in the glob
statement, and also then filter further using regular expressions to
choose only files matching more specific criteria. This should be
enough to get you started though.
>
hth
>
Jon.- Hide quoted text -
>
- Show quoted text -

Okies; postcoding before finishing your early morning coffee is not
the greatest of ideas!

I forgot to mention that glob will return pathnames as well. You'll
need to check that os.path.isfile( filename) returns True before
processing it...

Jon.

[Laurent Rahuel]

Re: python noob, multiple file i/o

Maybe the walk method in os module is what you need

http://docs.python.org/lib/os-file-dir.html

Regards

Jon Clements wrote:

On 16 Mar, 09:02, "Jon Clements" <jon...@googlem ail.comwrote:

>On 16 Mar, 03:56, "hiro" <Nun...@gmail.c omwrote:
>>
>>
>>
>>
>>

Hi there,

>>

I'm very new to python, the problem I need to solve is whats the "best/
simplest/cleanest" way to read in multiple files (ascii), do stuff to
them, and write them out(ascii).

>>

--
import os

>>

filePath = ('O:/spam/eggs/')
for file in os.listdir(file Path): #straight from docs
# iterate the function through all the files in the directory
# write results to separate files <- this is where I'm mostly
stuck.

>>

--
For clarity's sake, the file naming conventions for the files I'm
reading from are file.1.txt -file.nth.txt

>>

It's been a long day, i'm at my wits end, so I apologize in advance if
I'm not making much sense here.
syntax would also be great if you can share some recipes.

>>
>I'd try the glob module.
>>
>

>import glob
>>
># Get a list of filenames matching wildcard criteria
># (note that path is relative to working directory of program)
>matching_file_list = glob.glob('O:/spam/eggs/*.txt')
>>
># For each file that matches, open it and process it in some way...
>for filename in matching_file_list:
>    infile = file(filename)
>    outfile = file(filename + '.out','w')
>    # Process the input file line by line...
>    for line in infile:
>        pass # Do something more useful here, change line and write to
>outfile?
>    # Be explicit with file closures
>    outfile.close()
>    infile.close()
>

>>
>Of course, you can change the wild card criteria in the glob
>statement, and also then filter further using regular expressions to
>choose only files matching more specific criteria. This should be
>enough to get you started though.
>>
>hth
>>
>Jon.- Hide quoted text -
>>
>- Show quoted text -

>
Okies; postcoding before finishing your early morning coffee is not
the greatest of ideas!
>
I forgot to mention that glob will return pathnames as well. You'll
need to check that os.path.isfile( filename) returns True before
processing it...
>
Jon.

[Jordan]

Re: python noob, multiple file i/o

On Mar 16, 7:09 am, Laurent Rahuel <lrahuel.notg.. .@voila.frwrote :

>[QUOTE][QUOTE]
# Get a list of filenames matching wildcard criteria
# (note that path is relative to working directory of program)
matching_file_list = glob.glob('O:/spam/eggs/*.txt')

	

		

			
			

			
				Maybe the walk method in os module is what you needhttp://docs.python.org/lib/os-file-dir.html
>
Regards
>
Jon Clements wrote:

	

		

			
			

			
				 On 16 Mar, 09:02, "Jon Clements" <jon...@googlem  ail.comwrote:

	

		

			
			

			
				On 16 Mar, 03:56, "hiro" <Nun...@gmail.c  omwrote:
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				 Hi there,
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				 I'm very new to python, the problem I need to solve is whats the "best/
 simplest/cleanest" way to read in multiple files (ascii), do stuff to
 them, and write them out(ascii).
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				 --
 import os
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				 filePath = ('O:/spam/eggs/')
 for file in os.listdir(file  Path):   #straight from docs
     # iterate the function through all the files in the directory
     # write results to separate files  <- this is where I'm mostly
 stuck.
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				 --
 For clarity's sake, the file naming conventions for the files I'm
 reading from are file.1.txt -file.nth.txt
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				 It's been a long day, i'm at my wits end, so I apologize in advance if
 I'm not making much sense here.
 syntax would also be great if you can share some recipes.
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				I'd try the glob module.
			
		
	

			
		
	
>

	

		

			
			

			
				

	

		

			
			

			
				

	
	
import glob

			
		
	

			
		
	

			
		
	
[/QUOTE]
>[QUOTE][QUOTE]
# For each file that matches, open it and process it in some way...
for filename in matching_file_list:
    infile = file(filename)
    outfile = file(filename + '.out','w')
    # Process the input file line by line...
    for line in infile:
        pass # Do something more useful here, change line and write to
outfile?
    # Be explicit with file closures
    outfile.close()
    infile.close()
[/QUOTE][/QUOTE]

>

Of course, you can change the wild card criteria in the glob
statement, and also then filter further using regular expressions to
choose only files matching more specific criteria. This should be
enough to get you started though.

>

hth

>

Jon.- Hide quoted text -

>

- Show quoted text -

>

Okies; postcoding before finishing your early morning coffee is not
the greatest of ideas!

>

I forgot to mention that glob will return pathnames as well. You'll
need to check that os.path.isfile( filename) returns True before
processing it...

>

Jon.

[/QUOTE]

Also, leaving the format as .out is not necessarily convenient. You
had glob do a search for .txt, so how about doing:

Also, Python advises using open() over file() (although I admit to
using file() myself more often than not)

>>for filename in matching_file_l ist:
> infile = open(filename,' r') # add 'r' for clarity if nothing else
> outfile = open(filename[:-4] + '.out.txt','w') # assumes file ext of original file is .txt
> # Process the input file line by line...
> for line in infile:
> pass # do thing --you don't have to iterate line by line, if you specified what you wanted to do to each file we could probably help out here if you need it.
> # Be explicit with file closures
> outfile.close()
> infile.close()

Might also add some try/except statements to be safe ;).

Cheers,
Jordan

[7stud]

Re: python noob, multiple file i/o

The general idiom for altering lines in a file is to open the original
file and write the alterations to a temp file. After you are done
writing to the temp file, delete the original file, and change the
temp file name to the original file name.

If instead you were to read the whole file into a variable, and then
start overwriting the original file with the altered data, if your
program should happen to crash after writing one line to the file, all
the data in your variable would disappear into the ether, and your
file would only contain one line of data. You can imagine what that
would be like if you had lots of important data in the file.

Here's my attempt that incorporates a temp file:

-----
import os

filepath = "./change_files"

li = os.listdir(file path)
for name in li:
fullpath = filepath + "/" + name
if os.path.isfile( fullpath):
infile = open(fullpath, 'r')

lastDotPos = fullpath.rfind( ".")
fileName = fullpath[:lastDotPos]
ext = fullpath[lastDotPos:]
tempName = fileName + ext + ".temp"

outfile = open(tempName, "w")
for line in infile:
outfile.write(l ine + "altered\n" )
outfile.close()

os.remove(fullp ath)
os.rename(tempN ame, tempName[:-5])
---------------

I think it also needs some kind of check to make sure you have
permissions to delete the original file.

[7stud]

Re: python noob, multiple file i/o

On Mar 16, 9:38 am, "7stud" <bbxx789_0...@y ahoo.comwrote:

-----
import os
>
filepath = "./change_files"
>
li = os.listdir(file path)
for name in li:
fullpath = filepath + "/" + name
if os.path.isfile( fullpath):
infile = open(fullpath, 'r')
>
lastDotPos = fullpath.rfind( ".")
fileName = fullpath[:lastDotPos]
ext = fullpath[lastDotPos:]
tempName = fileName + ext + ".temp"
>
outfile = open(tempName, "w")
for line in infile:
outfile.write(l ine + "altered\n" )
outfile.close()
>
os.remove(fullp ath)
os.rename(tempN ame, tempName[:-5])
---------------
>

I did some unnecessary name manipulation in there. Try this:

filepath = "./change_files"

li = os.listdir(file path)
for name in li:
fullpath = filepath + "/" + name
if os.path.isfile( fullpath):
infile = open(fullpath, 'r')

tempName = fullpath + ".temp"
outfile = open(tempName, "w")
for line in infile:
outfile.write(l ine + "altered\n" )
outfile.close()

os.remove(fullp ath)
os.rename(tempN ame, tempName[:-5])

[hiro]

Re: python noob, multiple file i/o

Thanks a lot for the help guys, I'm at work right now and I will go
over your suggestions one by one this weekend. Being more alert now,
taking a look at the examples you posted, I now see how to approach
this problem. The thing with python that I'm starting to realize is
that there are a million different ways to approach a problem, so I
find it great for experimenting (when time allows) yet very
challenging to choose an approach.

Cheers,

h.

[hiro]

Re: python noob, multiple file i/o

Thanks a lot for the help guys, I'm at work right now and I will go
over your suggestions one by one this weekend. Being more alert now,
taking a look at the examples you posted, I now see how to approach
this problem. The thing with python that I'm starting to realize is
that there are a million different ways to approach a problem, so I
find it great for experimenting (when time allows) yet very
challenging to choose an approach.

Cheers,

h.