Is there a way of setting O_DIRECT on a preexisting file like sys.stdin?
Does C allow this sort of thing?
Thanks!
-
Is there a way of setting O_DIRECT on a preexisting file like sys.stdin?
Does C allow this sort of thing?
Thanks!
-
Re: O_DIRECT on stdin?
>Is there a way of setting O_DIRECT on a preexisting file like sys.stdin?[color=blue]
>
>Does C allow this sort of thing?[/color]
There is no O_DIRECT or fcntl() in Standard C.
fcntl() operates on an open file descriptor, and the file descriptor
for stdin is 0. Two calls to fcntl(), one with F_GETFL and one
with F_SETFL, would do what you want.
I'm not sure why you want to do that, though. It's not going to
get you character-at-a-time I/O, if that's what you want.
Gordon L. Burditt
-
Re: O_DIRECT on stdin?
I think this is fcntl(..., F_SETFL, ...), so something like
import os, fcntl, sys
flags = fcntl.fcntl(sys .stdin.fileno() , fcntl.F_GETFL)
flags |= os.O_DIRECT
fcntl.fcntl(sys .stdin.fileno() , fcntl.F_SETFL, flags)
Jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
iD8DBQFDb7CcJd0 1MZaTXX0RAi5+AJ 0eeXXROgvPtbcg6 br8Zf/KsGPAxgCeLFb0
xNIG26j8JMJIecG iNWmcU0E=
=2LQx
-----END PGP SIGNATURE-----
Comment
-
Re: O_DIRECT on stdin?
On Mon, 07 Nov 2005 19:48:47 +0000, Gordon Burditt wrote:
[color=blue][color=green]
>> [quoted text muted][/color]
>
> There is no O_DIRECT or fcntl() in Standard C.
>
> fcntl() operates on an open file descriptor, and the file descriptor
> for stdin is 0. Two calls to fcntl(), one with F_GETFL and one
> with F_SETFL, would do what you want.
>
> I'm not sure why you want to do that, though. It's not going to
> get you character-at-a-time I/O, if that's what you want.
>
> Gordon L. Burditt[/color]
I want to be able to read a HUGE file without having such a negative
impact on the system's buffer cache.
I'm trying:
if hasattr(os, 'O_DIRECT'):
try:
flags = fcntl.fcntl(sys .stdin.fileno() , fcntl.F_GETFL)
flags |= os.O_DIRECT
fcntl.fcntl(sys .stdin.fileno() , fcntl.F_SETFL, flags)
except:
sys.stderr.writ e('Setting O_DIRECT on stdin attempted but failed\n')
else:
sys.stderr.writ e('Setting O_DIRECT on stdin succeeded :)\n')
....but while this code doesn't error out, I get:
seki-root> reblock -e $[1024*1024*80] $[1024*1024] 300 < /dev/sda1 > /dev/null
stdin seems seekable, but file length is 0 - no exact percentages
Estimated filetransfer size is 85899345920 bytes
Estimated percentages will only be as accurate as your size estimate
Setting O_DIRECT on stdin succeeded :)
Traceback (most recent call last):
File "/Dcs/seki/strombrg/bin/reblock", line 276, in ?
main()
File "/Dcs/seki/strombrg/bin/reblock", line 222, in main
block = os.read(0,block size)
OSError: [Errno 22] Invalid argument
Mon Nov 07 12:25:53
....but if I comment out the fcntl/O_DIRECT code, then the same thing works
well.
Any other ideas folks?
Thanks!
Comment
-
Re: O_DIRECT on stdin?
>I want to be able to read a HUGE file without having such a negative[color=blue]
>impact on the system's buffer cache.
>
>I'm trying:
>
> if hasattr(os, 'O_DIRECT'):
> try:
> flags = fcntl.fcntl(sys .stdin.fileno() , fcntl.F_GETFL)
> flags |= os.O_DIRECT
> fcntl.fcntl(sys .stdin.fileno() , fcntl.F_SETFL, flags)
> except:
> sys.stderr.writ e('Setting O_DIRECT on stdin attempted but failed\n')
> else:
> sys.stderr.writ e('Setting O_DIRECT on stdin succeeded :)\n')
>
>...but while this code doesn't error out, I get:
>
>seki-root> reblock -e $[1024*1024*80] $[1024*1024] 300 < /dev/sda1 > /dev/null
>stdin seems seekable, but file length is 0 - no exact percentages
>Estimated filetransfer size is 85899345920 bytes
>Estimated percentages will only be as accurate as your size estimate
>Setting O_DIRECT on stdin succeeded :)
>Traceback (most recent call last):
> File "/Dcs/seki/strombrg/bin/reblock", line 276, in ?
> main()
> File "/Dcs/seki/strombrg/bin/reblock", line 222, in main
> block = os.read(0,block size)
>OSError: [Errno 22] Invalid argument
>Mon Nov 07 12:25:53
>
>...but if I comment out the fcntl/O_DIRECT code, then the same thing works
>well.
>
>Any other ideas folks?[/color]
Does O_DIRECT perhaps invoke some of the restrictions of "raw"
device files, where the current offset and transfer size must be a
multiple of some block size? (I don't see any mention of that in
FreeBSD's documentation.) What is the value of blocksize at the
time of the traceback above? I suggest keeping it well under 2G.
Gordon L. Burditt
Comment
-
Re: O_DIRECT on stdin?
On Mon, 07 Nov 2005 20:42:50 +0000, Gordon Burditt wrote:
[color=blue][color=green]
>> [quoted text muted][/color]
>
> Does O_DIRECT perhaps invoke some of the restrictions of "raw"
> device files, where the current offset and transfer size must be a
> multiple of some block size? (I don't see any mention of that in
> FreeBSD's documentation.) What is the value of blocksize at the
> time of the traceback above? I suggest keeping it well under 2G.
>
> Gordon L. Burditt[/color]
I'm not aware of such a restriction on O_DIRECT, but then I just learned
of O_DIRECT's existence earlier today. :)
The value of blocksize at the time of the error should be 1 megabyte.
Comment
-
Re: O_DIRECT on stdin?
Here's some text from my open(2) manpage:
Transfer sizes, and the alignment of user buffer and file offset mustall
be multiples of the logical block size of the file system.
It's unlikely that in practice you can get Python's sys.stdin.read( ) or
os.read() to reliably use a buffer that fits the alignment restriction.
However, a small "C" extension could provide something like os.read() which
*does* meet the restriction. The meat of it would look something like
#define PAGESIZE 4096 // a guess which may be right for x86 linux
#define REQUESTSIZE 1048576
ssize_t result;
char *buf, *base;
PyObject *pyresult;
buf = malloc(bufsize, REQUESTSIZE + PAGESIZE - 1);
base = round_up(buf, PAGESIZE);
result = read(0, base, REQUESTSIZE);
if(result == -1) {
set python error from errno
pyresult = NULL;
goto DONE;
}
pyresult = PyString_FromSt ringAndSize(bas e, result);
DONE:
free(buf);
return pyresult;
Here's a clever but untested "C" macro that claims to implement round_up:
#define round_up(amount , align) ((((amount) - 1) | ((align) - 1)) +1)
Jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
iD8DBQFDb8iGJd0 1MZaTXX0RAm/AAKCjJmBlQ/Lzu5kEnnA/GzqLeeE03QCffJW P
0T/vsxfB6dLlr4kvwa OtJjE=
=3Pyy
-----END PGP SIGNATURE-----
Comment
-
Re: O_DIRECT on stdin?
In article <mailman.237.11 31399306.18701. python-list@python.org >,
jepler@unpython ic.net wrote:
[color=blue]
> Here's some text from my open(2) manpage:
> Transfer sizes, and the alignment of user buffer and file offset must
> all
> be multiples of the logical block size of the file system.[/color]
Does that apply in the example he gave, < /dev/sda1 ?
It seems to me this would not go through any filesystem anyway.
That might account for the "invalid argument" error, but at any
rate it would be irrelevant.
Plus it doesn't seem to score very high on portability, according
to the Linux man page I'm looking at -- apparently not a POSIX
or any such standard, just borrowed from Irix in recent Linux
versions, and FreeBSD with slightly different behavior. Don't
see any trace of it in NetBSD, MacOS X.
[color=blue]
> It's unlikely that in practice you can get Python's sys.stdin.read( ) or
> os.read() to reliably use a buffer that fits the alignment restriction.[/color]
Though of course os.read() would eliminate one layer of buffering
altogether. Might be worth a try.
Donn Cave, donn@u.washingt on.edu
Comment
-
Re: O_DIRECT on stdin?
"Gordon Burditt" <gordon@hammy.b urditt.org> wrote in message
news:11mvf2a2b8 nf54b@corp.supe rnews.com...[color=blue][color=green]
> >I want to be able to read a HUGE file without having such a negative
> >impact on the system's buffer cache.[/color][/color]
[snip][color=blue]
> Does O_DIRECT perhaps invoke some of the restrictions of "raw"
> device files, where the current offset and transfer size must be a
> multiple of some block size?[/color]
Very likely. It is also likely that the same applies to the destination (ie
memory) address.
Alex
Comment
Comment