exclude binary files from os.walk

[rbt]

Is there an easy way to exclude binary files (I'm working on Windows XP)
from the file list returned by os.walk()?

Also, when reading files and you're unsure as to whether or not they are
ascii or binary, I've always thought it safer to 'rb' on the read, is
this correct... and if so, what's the reasoning behind this? Again all
of this pertains to files on Windows XP and Python 2.4

Many thanks!

[Grant Edwards]

Re: exclude binary files from os.walk

On 2005-01-26, rbt <rbt@athop1.ath .vt.edu> wrote:
[color=blue]
> Is there an easy way to exclude binary files (I'm working on
> Windows XP) from the file list returned by os.walk()?[/color]

Sure, assuming you can provide a rigorous definition of 'binary
files'. :)
[color=blue]
> Also, when reading files and you're unsure as to whether or
> not they are ascii or binary, I've always thought it safer to
> 'rb' on the read, is this correct...[/color]

That depends on what you want. Adding a 'b' will disable the
cr/lf handling. Not sure what else it does.
[color=blue]
> and if so, what's the reasoning behind this?[/color]

Behind what?
[color=blue]
> Again all of this pertains to files on Windows XP and Python 2.4[/color]

--
Grant Edwards grante Yow! Now that I have my
at "APPLE", I comprehend COST
visi.com ACCOUNTING!!

[rbt]

Re: exclude binary files from os.walk

Grant Edwards wrote:[color=blue]
> On 2005-01-26, rbt <rbt@athop1.ath .vt.edu> wrote:
>
>[color=green]
>>Is there an easy way to exclude binary files (I'm working on
>>Windows XP) from the file list returned by os.walk()?[/color]
>
>
> Sure, assuming you can provide a rigorous definition of 'binary
> files'. :)[/color]

non-ascii

[Larry Bates]

Re: exclude binary files from os.walk

There's no definitive way of telling a file is
"non-ascii". Bytes in a binary file define
perfectly good ascii characters. Windows
depends on file extensions to try to keep track
of the "type" of data in a file, but that isn't
foolproof. I can rename a plain ascii file with
a .EXE extension.

We could be of more help, if you would take the
time to explain a little about what you are trying
to do.

Larry Bates

rbt wrote:[color=blue]
> Grant Edwards wrote:
>[color=green]
>> On 2005-01-26, rbt <rbt@athop1.ath .vt.edu> wrote:
>>
>>[color=darkred]
>>> Is there an easy way to exclude binary files (I'm working on
>>> Windows XP) from the file list returned by os.walk()?[/color]
>>
>>
>>
>> Sure, assuming you can provide a rigorous definition of 'binary
>> files'. :)[/color]
>
>
> non-ascii[/color]

[sp1d3rx@gmail.com]

Re: exclude binary files from os.walk

you might want to look up the 'isascii' function...
i.e. - can be represented using just 7-bits.

[Dan Perl]

Re: exclude binary files from os.walk


"rbt" <rbt@athop1.ath .vt.edu> wrote in message
news:ct94j7$15a $1@solaris.cc.v t.edu...[color=blue]
> Is there an easy way to exclude binary files (I'm working on Windows XP)
> from the file list returned by os.walk()?
>
> Also, when reading files and you're unsure as to whether or not they are
> ascii or binary, I've always thought it safer to 'rb' on the read, is this
> correct... and if so, what's the reasoning behind this? Again all of this
> pertains to files on Windows XP and Python 2.4[/color]

Please clarify: is your question about identifying binary (non-ascii) files
or about using os.walk?

[Grant Edwards]

Re: exclude binary files from os.walk

On 2005-01-26, Larry Bates <lbates@syscono nline.com> wrote:
[color=blue]
> There's no definitive way of telling a file is "non-ascii".
> Bytes in a binary file define perfectly good ascii characters.[/color]

As long as bit 7 is a 0.

Traditional ASCII only allows/defines the values 0x00 through
0x7f. If that's what is meant by "ASCII", then a file
containting bytes greater than 0x7F is not ASCII.

If all bytes are 0x7F or below, the file _may_ be ASCII, but
there's now way to tell if it _is_ ASCII unless you ask the
creator of the file. It could be Baudot or some other encoding
that doesn't use bit 7. Or, it could just be binary data that
happens to have bit 7 == 0.
[color=blue]
> We could be of more help, if you would take the time to
> explain a little about what you are trying to do.[/color]

Yup.

--
Grant Edwards grante Yow! Now, let's SEND OUT
at for QUICHE!!
visi.com

[Craig Ringer]

Re: exclude binary files from os.walk

On Wed, 2005-01-26 at 17:32 -0500, rbt wrote:[color=blue]
> Grant Edwards wrote:[color=green]
> > On 2005-01-26, rbt <rbt@athop1.ath .vt.edu> wrote:
> >
> >[color=darkred]
> >>Is there an easy way to exclude binary files (I'm working on
> >>Windows XP) from the file list returned by os.walk()?[/color]
> >
> >
> > Sure, assuming you can provide a rigorous definition of 'binary
> > files'. :)[/color]
>
> non-ascii[/color]

That's not really safe when dealing with utf-8 files though, and IIRC
with UCS2 or UCS4 as well. The Unicode BOM its self might (I'm not sure)
qualify as ASCII.

--
Craig Ringer

[Mark McEahern]

Re: exclude binary files from os.walk

The OP wrote:
[color=blue]
> Is there an easy way to exclude binary files (I'm working on Windows[/color]
XP) from the file list returned by os.walk()?

Sure, piece of cake:

#!/usr/bin/env python

import os

def textfiles(path) :
include = ('.txt', '.csv',)
for root, dirs, files in os.walk(path):
for name in files:
prefix, ext = os.path.splitex t(name)
if ext.lower() not in include:
continue
filename = os.path.join(ro ot, name)
yield filename

path = os.getcwd()
for name in textfiles(path) :
print name

;-)

// m

[Alex Martelli]

Re: exclude binary files from os.walk

rbt <rbt@athop1.ath .vt.edu> wrote:
[color=blue]
> Grant Edwards wrote:[color=green]
> > On 2005-01-26, rbt <rbt@athop1.ath .vt.edu> wrote:
> >[color=darkred]
> >>Is there an easy way to exclude binary files (I'm working on
> >>Windows XP) from the file list returned by os.walk()?[/color]
> >
> > Sure, assuming you can provide a rigorous definition of 'binary
> > files'. :)[/color]
>
> non-ascii[/color]

The only way to tell for sure if a file contains only ASCII characters
is to read the whole file and check. You _are_, however, using a very
strange definition of "binary". A file of text in German, French or
Italian, for example, is likely to be one you'll define as "binary" --
just as soon as it contains a vowel with accent or diaeresis, for
example. On the other hand, you want to consider "non-binary" a file
chock full of hardly-ever-used control characters, just because the
American Standard Code for Information Interchange happened to
standardize them once upon a time? Most people's intuitive sense of
what "binary" means would rebel against both of these choices, I think;
calling a file "binary" because its contents are, say, the string
'El perro de aguas español.\n' (the n-with-tilde in "español"
disqualifies it from being ASCII), while another whose contents are 32
bytes all made up of 8 zero bits each (ASCII 'NUL' characters) is to be
considered "non-binary".

In any case, since you need to open and read all the files to check them
for "being binary", either by your definition or whatever heuristics you
might prefer, you would really not ``excluded them from os.walk'', but
rather filter os.walk's results by these criteria.


Alex

[Alex Martelli]

Re: exclude binary files from os.walk

Craig Ringer <craig@postnews papers.com.au> wrote:
[color=blue]
> That's not really safe when dealing with utf-8 files though, and IIRC
> with UCS2 or UCS4 as well. The Unicode BOM its self might (I'm not sure)
> qualify as ASCII.[/color]

Nope, both bytes in the BOM have the high-order bit set -- they're 0xFF
and 0xFE -- so they definitely don't qualify as ASCII.


Alex

[Bengt Richter]

Re: exclude binary files from os.walk

On Wed, 26 Jan 2005 18:25:09 -0500, "Dan Perl" <danperl@rogers .com> wrote:
[color=blue]
>
>"rbt" <rbt@athop1.ath .vt.edu> wrote in message
>news:ct94j7$15 a$1@solaris.cc. vt.edu...[color=green]
>> Is there an easy way to exclude binary files (I'm working on Windows XP)
>> from the file list returned by os.walk()?
>>
>> Also, when reading files and you're unsure as to whether or not they are
>> ascii or binary, I've always thought it safer to 'rb' on the read, is this
>> correct... and if so, what's the reasoning behind this? Again all of this
>> pertains to files on Windows XP and Python 2.4[/color]
>
>Please clarify: is your question about identifying binary (non-ascii) files
>or about using os.walk?
>
>[/color]
I have a feeling it's about walking directories and identifying which files
to should be "cooked" (to normalize line endings when opened and read).

Regards,
Bengt Richter