Signed and unsigned int

[jacob navia]

A signed int can contain up to 2Gig, 2 147 483 648, to be exact.

Since The Mars rovers landed, I have been storing the
photographs in two directories, Spirit and Opportunity. I had
more than 18 000 files in a single directory. Without being
aware of it, I crossed the 2 147 483 648 border last week.

Nothing happens, if you do not attempt to read all files in the directory,
or even worst, copy them to another drive.

Worried by mysterious disk errors, I tried to save my work. When making
the copy however, there was an error when the copy arrived at an image
and could not read it.

Foolish me, I tried to copy the file again.

That was it: the disk started to make a periodic mechanical noise, and
it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!! !!!!!!!!!!

Why?

When a signed integer is increased beyond 2147483648, it becomes NEGATIVE.
This means that the system will issue a NONSENSE movemevent order to the
read heads, destroying the disk instantly.

I was lucky. I had a full backup of the mars data in my linux system. Ahh
Microsoft.
I fired up the linux machine running ext2 file system. I issued the order to
copy
the files, and started doing other things during the copy.

When the amount of data transmitted arrived at approx 2GB, I heared with
horror that
the disk started doing the SAME repeating mechanical noise and my linux
system
WAS GONE, I HAVE LOST several months of work without any means of getting
my data back.

Signed integer can contain up to 2147483648 bytes. Not a single byte more.

I have developed a compile time switch to check for overflows within
lcc-win32 and
posted here a message, several weeks ago. Nobody cared to answer.

SHIT !!!!!!!!!!!

C is a nice language in which to write file systems. But IT WOULD BE BETTER
TO
BE CAREFUL WITH THOSE "int" s OK?

You do not believe me?

Try it. Make several directories with several thousand files of 100-200K
each, until
you get more than 2GB.

But backup your drive first...

jacob

[Christian Bau]

Re: Signed and unsigned int

In article <cfbegn$r34$1@n ews-reader4.wanadoo .fr>,
"jacob navia" <jacob@jacob.re mcomp.fr> wrote:
[color=blue]
> Try it. Make several directories with several thousand files of 100-200K
> each, until
> you get more than 2GB.
>
> But backup your drive first...[/color]

A digital video camera records exactly 3.6 million bytes per seoncd.
That is 216 million bytes per minute, or 12,960,000,000 byte per hour. I
have directories on my harddisk with about 13 Gigabyte of contents, my
neighbour has about 30 Gigabyte in a single directory - just 2 1/2 hours
of digital video.

Both machines are Macs using the HFS+ filesystem, but I would think that
there is video editing software for Windows and Linux, and I am sure it
must be able to handle similar amounts of data.

[Emmanuel Delahaye]

Re: Signed and unsigned int

jacob navia wrote on 10/08/04 :[color=blue]
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.[/color]

Of course you meant

"A signed long can contain at least to 2Gig, 2 147 483 648, to be
exact."

or

"On my machine, a signed int can contain up to 2Gig, 2 147 483 648, to
be exact."

--
Emmanuel
The C-FAQ: http://www.eskimo.com/~scs/C-faq/faq.html

"C is a sharp tool"

[Eric Sosman]

[OT] Re: Signed and unsigned int

jacob navia wrote:[color=blue]
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.[/color]

"Exact" if you ignore the off-by-one error ...
[color=blue]
> Since The Mars rovers landed, I have been storing the
> photographs in two directories, Spirit and Opportunity. I had
> more than 18 000 files in a single directory. Without being
> aware of it, I crossed the 2 147 483 648 border last week.
> [... and two different O/Ses trashed the data].
>
> C is a nice language in which to write file systems. But IT WOULD BE BETTER
> TO
> BE CAREFUL WITH THOSE "int" s OK?[/color]

Jacob, while I'm genuinely sorry for your data loss I think
you may have fingered the wrong culprit. It seems unlikely that
any file system nowadays would have a 31-bit limit built in; disks
have been larger than 2GB for a long time now. In fact, disks have
been larger than 4GB for a long time; you'd probably need to look
long and hard to find a drive that small today. And if the file
systems using these larger drives had been limited by 31-bit or
even 32-bit byte counts, plenty of people other than you would
have noticed ...
[color=blue]
> You do not believe me?
>
> Try it. Make several directories with several thousand files of 100-200K
> each, until
> you get more than 2GB.[/color]

Well, I'm using files of 50-100MB instead, but I hope you'll
accept the (slightly edited) evidence anyhow:

Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c14t40d0s6 356770888 37071528 316131656 11% (withheld)
/dev/dsk/c14t40d1s6 356770888 37140512 316062672 11%
/dev/dsk/c13t40d0s6 356770888 37150672 316052512 11%
/dev/dsk/c16t40d0s6 356770888 37206864 315996320 11%
/dev/dsk/c13t40d1s6 356770888 37356592 315846592 11%
/dev/dsk/c15t40d0s6 356770888 37412760 315790424 11%
/dev/dsk/c16t40d1s6 356770888 53911776 299291408 16%
/dev/dsk/c15t40d1s6 356770888 54248872 298954312 16%

I'd bet that the 31-bit limit (the number in your sad tale
is too coincidental to ignore) has nothing to do with the file
system. It may, perhaps, have something to do with the utility
you were using to mass-copy all that data.

My condolences.

--
Eric.Sosman@sun .com

[Gordon Burditt]

Re: Signed and unsigned int

>A signed int can contain up to 2Gig, 2 147 483 648, to be exact.

I know of no system where this is the limit. A 32-bit signed int
cannot contain 2 147 483 648. It can contain 2 147 483 647. It
is not guaranteed that a signed int has more than 16 bits, nor is
there anything preventing it from having 47 or 128 bits.
[color=blue]
>Since The Mars rovers landed, I have been storing the
>photographs in two directories, Spirit and Opportunity. I had
>more than 18 000 files in a single directory. Without being
>aware of it, I crossed the 2 147 483 648 border last week.
>
>Nothing happens, if you do not attempt to read all files in the directory,
>or even worst, copy them to another drive.[/color]

Most of the problems I have heard of involve having a SINGLE FILE
containing more than 2GB, or 4GB. I hope there is not much software
in use where a whole disk drive is constrained by size limits of
2GB or 4GB. I don't understand why there is a limit on reading a
bunch of files in a directory (unless you were copying them into a
single file, e.g. with tar or cpio) rather than a limit on individual
file size or disk partition size.

A reasonable OS will not let you write a file beyond the limit,
rather than corrupting the file system, whatever that limit is,
even if it's something huge like 512YB.
[color=blue]
>Worried by mysterious disk errors, I tried to save my work. When making
>the copy however, there was an error when the copy arrived at an image
>and could not read it.
>
>Foolish me, I tried to copy the file again.
>
>That was it: the disk started to make a periodic mechanical noise, and
>it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!! !!!!!!!!!![/color]

I'm sorry about your lost files but I'm not entirely convinced that
copying too much at once is the reason you lost them.
[color=blue]
>Why?
>
>When a signed integer is increased beyond 2147483648, it becomes NEGATIVE.
>This means that the system will issue a NONSENSE movemevent order to the
>read heads, destroying the disk instantly.[/color]

A reasonable OS will not issue such an order to the disk drive.
The driver should range-check the values it is computing and object
if the values are out of range. So, for that matter, should the
disk drive. Disk drives nowadays are much smarter than the old
mechanical floppy drives where you could keep stepping and slam the
heads into things.
[color=blue]
>I was lucky. I had a full backup of the mars data in my linux system. Ahh
>Microsoft.
>I fired up the linux machine running ext2 file system. I issued the order to
>copy
>the files, and started doing other things during the copy.
>
>When the amount of data transmitted arrived at approx 2GB, I heared with
>horror that
>the disk started doing the SAME repeating mechanical noise and my linux
>system
>WAS GONE, I HAVE LOST several months of work without any means of getting
>my data back.
>
>Signed integer can contain up to 2147483648 bytes. Not a single byte more.
>
>I have developed a compile time switch to check for overflows within
>lcc-win32 and
>posted here a message, several weeks ago. Nobody cared to answer.
>
>SHIT !!!!!!!!!!!
>
>C is a nice language in which to write file systems. But IT WOULD BE BETTER
>TO
>BE CAREFUL WITH THOSE "int" s OK?
>
>You do not believe me?
>
>Try it. Make several directories with several thousand files of 100-200K
>each, until
>you get more than 2GB.[/color]

I routinely make single files that are greater than 4GB on FreeBSD.
(The size of a burnable DVD image is about 4.7GB if you pack things
to use the maximum capacity. I'm using DVDs to store DATA, not
video or audio. The number goes up if your DVD supports double-sided
or blue-laser DVDs) Yes, I've got some directory trees somewhat
like what you describe, only the total is 22GB (most of it downloaded
with a single multi-file FTP command). Yes, there's a backup of
it on another system as a single 22GB tar file. I need to split
it up into chunks small enough to burn on DVDs. (actually, I've
done this several times, but I don't like how the files end up
getting split - I want related files on the same disk. I do need
to check the program that does the splitting, but I think it's using
off_t's for holding file lengths (64 bits on FreeBSD)). After I do
that, the program that makes a DVD image is essentially a copy
program that puts all the files in one big file (with assorted
headers & stuff). I have had no problems.

I believe I have done this on Linux also, even old versions of Linux
(think TiVo's OS - was that built in 1996?) only supporting LBA32
and not LBA48 (meaning Linux won't support DRIVES larger than 137GB
or so). Modern versions of Linux don't have that limit.

Gordon L. Burditt

[Keith Thompson]

Re: Signed and unsigned int

"jacob navia" <jacob@jacob.re mcomp.fr> writes:[color=blue]
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.[/color]

Typically 2 147 483 647, to be even more exact, but of course it can
be any value greater than or equal to 32767.
[color=blue]
> Since The Mars rovers landed, I have been storing the
> photographs in two directories, Spirit and Opportunity. I had
> more than 18 000 files in a single directory. Without being
> aware of it, I crossed the 2 147 483 648 border last week.
>
> Nothing happens, if you do not attempt to read all files in the directory,
> or even worst, copy them to another drive.
>
> Worried by mysterious disk errors, I tried to save my work. When making
> the copy however, there was an error when the copy arrived at an image
> and could not read it.
>
> Foolish me, I tried to copy the file again.
>
> That was it: the disk started to make a periodic mechanical noise, and
> it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!! !!!!!!!!!!
>
> Why?[/color]

Because there's a bug in your OS's filesystem code. (Do you know, or
are you assuming, that that code was written in C?)

[...][color=blue]
> I have developed a compile time switch to check for overflows within
> lcc-win32 and posted here a message, several weeks ago. Nobody cared
> to answer.[/color]

Would this have prevented your problem? Detecting overflow is only
part of the problem; the programmer has to decide what to do once the
overflow is detected. Using a compiler that traps overflows might
have caused whatever program you were using (or the OS itself) to
crash rather than continue with invalid data, which is a bit of an
improvement but not enough of one.
[color=blue]
> C is a nice language in which to write file systems. But IT WOULD BE
> BETTER TO BE CAREFUL WITH THOSE "int" s OK?[/color]

"Be careful" is always good advice, but you have to know *how* to be
careful. If you want to count more than 2**31-1 of something, you
just need to use something bigger than a 32-bit signed type.
Detecting overflow is a good way to detect bugs, but it's seldom a
good way to correct them; it's rare to want to an overflow handler to
be executed during the normal execution of a program.

<OT>I'm surprised that a Linux ext2 filesystem would behave the way
you describe, but of course this isn't the place to discuss the
specifics.</OT>

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Keith Thompson]

Re: Signed and unsigned int

Christian Bau <christian.bau@ cbau.freeserve. co.uk> writes:[color=blue]
> In article <cfbegn$r34$1@n ews-reader4.wanadoo .fr>,
> "jacob navia" <jacob@jacob.re mcomp.fr> wrote:
>[color=green]
> > Try it. Make several directories with several thousand files of 100-200K
> > each, until
> > you get more than 2GB.
> >
> > But backup your drive first...[/color]
>
> A digital video camera records exactly 3.6 million bytes per seoncd.
> That is 216 million bytes per minute, or 12,960,000,000 byte per hour. I
> have directories on my harddisk with about 13 Gigabyte of contents, my
> neighbour has about 30 Gigabyte in a single directory - just 2 1/2 hours
> of digital video.
>
> Both machines are Macs using the HFS+ filesystem, but I would think that
> there is video editing software for Windows and Linux, and I am sure it
> must be able to handle similar amounts of data.[/color]

<WAY_OT>
For Windows, FAT32 vs. NTFS might be significant.
</WAY_OT>

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Joe Wright]

Re: Signed and unsigned int

jacob navia wrote:[color=blue]
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.
>
> Since The Mars rovers landed, I have been storing the
> photographs in two directories, Spirit and Opportunity. I had
> more than 18 000 files in a single directory. Without being
> aware of it, I crossed the 2 147 483 648 border last week.
>
> Nothing happens, if you do not attempt to read all files in the directory,
> or even worst, copy them to another drive.
>
> Worried by mysterious disk errors, I tried to save my work. When making
> the copy however, there was an error when the copy arrived at an image
> and could not read it.
>
> Foolish me, I tried to copy the file again.
>
> That was it: the disk started to make a periodic mechanical noise, and
> it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!! !!!!!!!!!!
>
> Why?
>
> When a signed integer is increased beyond 2147483648, it becomes NEGATIVE.
> This means that the system will issue a NONSENSE movemevent order to the
> read heads, destroying the disk instantly.
>
> I was lucky. I had a full backup of the mars data in my linux system. Ahh
> Microsoft.
> I fired up the linux machine running ext2 file system. I issued the order to
> copy
> the files, and started doing other things during the copy.
>
> When the amount of data transmitted arrived at approx 2GB, I heared with
> horror that
> the disk started doing the SAME repeating mechanical noise and my linux
> system
> WAS GONE, I HAVE LOST several months of work without any means of getting
> my data back.
>
> Signed integer can contain up to 2147483648 bytes. Not a single byte more.
>
> I have developed a compile time switch to check for overflows within
> lcc-win32 and
> posted here a message, several weeks ago. Nobody cared to answer.
>
> SHIT !!!!!!!!!!!
>
> C is a nice language in which to write file systems. But IT WOULD BE BETTER
> TO
> BE CAREFUL WITH THOSE "int" s OK?
>
> You do not believe me?
>
> Try it. Make several directories with several thousand files of 100-200K
> each, until
> you get more than 2GB.
>
> But backup your drive first...
>
> jacob
>
>
>
>
>
>[/color]
The largest signed int on your system is 2147483647, not 2147483648.
Navia should know this. 2^31-1 is a magic number on virtually all
32-bit computer architectures.

01111111 11111111 11111111 11111111 is magic. Magic +1 is..

10000000 00000000 00000000 00000000 and too much magic.

Having said that, I'm surprised and saddened that both Windows and
Linux would actually fall over past the upper bound. I would have
expected something more graceful.
--
Joe Wright mailto:joewwrig ht@comcast.net
"Everything should be made as simple as possible, but not simpler."
--- Albert Einstein ---

[CBFalconer]

Re: Signed and unsigned int

jacob navia wrote:[color=blue]
>
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.[/color]

No, a signed int can contain values from INT_MIN through INT_MAX,
if you #include <limits.h>. Those values must be at least -32767
through 32767.

If you need values from 0 to 2,147,483,648, use an unsigned long
and check for overflow yourself.

--
"Churchill and Bush can both be considered wartime leaders, just
as Secretariat and Mr Ed were both horses." - James Rhodes.
"A man who is right every time is not likely to do very much."
- Francis Crick, co-discover of DNA

[Keith Thompson]

Re: [OT] Signed and unsigned int

Joe Wright <joewwright@com cast.net> writes:
[...][color=blue]
> Having said that, I'm surprised and saddened that both Windows and
> Linux would actually fall over past the upper bound. I would have
> expected something more graceful.[/color]

Note the "[OT]" tag.

I'd be very surprised if the failures Jacob encountered were caused by
something as simple as a signed 32-bit integer overflow in the
filesystem implementation. I'd be especially surprised by this for
the Linux system; for Windows, I could almost believe it for FAT32,
but not for NTFS. (Not that I have any particular expertise for any
of these, but I've used them.) The failure at 2 gigabytes is
suspicious, but I'd expect any of the relevant filesystems to handle
at least that much data in a partition, and I wouldn't expect any them
to have a limitation that depends on the total size of the files in a
single directory.

Perhaps there's a bug in some program he was using to copy the files
-- though it's still surprising that such a bug would trash the target
Linux system.

Never underestimate the destructive power of an unlucky coincidence.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Richard Bos]

Re: Signed and unsigned int

Keith Thompson <kst-u@mib.org> wrote:
[color=blue]
> Christian Bau <christian.bau@ cbau.freeserve. co.uk> writes:[color=green]
> > Both machines are Macs using the HFS+ filesystem, but I would think that
> > there is video editing software for Windows and Linux, and I am sure it
> > must be able to handle similar amounts of data.[/color]
>
> <WAY_OT>
> For Windows, FAT32 vs. NTFS might be significant.
> </WAY_OT>[/color]

Nope. Not on this machine. I don't know _what_ Jacob's been up to, but
blaming his gaffes on C's ints sounds rather cheap to me.

Richard

[jacob navia]

Re: Signed and unsigned int

Sorry for not replying but I had to rebuild my machine
and mail sin't there yet.

I repeat:

The bug is not about SINGLE files bigger than 2GB.

It is when the contents of a directory exceed 2GB. Each
of the files is between 30-300K.

Maybe it is relevant to the bug that the upper and lower
directories are also quite big (0.8-1.5GB)

I repeat that the bug is related to the 2GB limit since it appeared
when I crossed it, around monday, when I updated
the directory from the NASA site.

Sorry but I do not know if I can answer more, but it
is VERY EASY TO REPRODUCE IT.

Just make a program that makes random files of
33-300K until you get 3GB, then do some directories and
make files.

I would be *very* interested in hearing from you!

jacob

[boa]

OT: File systems and lost files. (Was Signed and unsigned int)

jacob navia wrote:
[color=blue]
> Sorry but I do not know if I can answer more, but it
> is VERY EASY TO REPRODUCE IT.[/color]

Ok...
[color=blue]
>
> Just make a program that makes random files of
> 33-300K until you get 3GB, then do some directories and
> make files.[/color]

So I did, 34701 files in one directory, with a total size of 5.7G. Did a
umount+fsck+mou nt. No errors. File system was ext2 on a linux 2.4.25
running under VMWare.

boa@home

[Gordon Burditt]

Re: Signed and unsigned int

>It is when the contents of a directory exceed 2GB. Each[color=blue]
>of the files is between 30-300K.[/color]

What do you think is overflowing here? What is it that even bothers
to compute the total size of files in a directory? (Well, the
commands "ls" and "du" might, but that's not what you were running
at the time of the problem, was it?) To the best of my knowledge,
*NO* file I/O operation (open, close, read, write, seek, or whatever,
and no, those are not references to C function names) is the slightest
bit interested in the total amount of space taken by files in the
same directory as the file in question, and it does not bother to
compute such a number (so it can't overflow).

Could the problem be related to the total size of files *IN THE FILESYSTEM*?
How big is this filesystem? How much in files, total were there in
there? (Could you have just crossed 4GB for this number?)
Could the problem be related to RUNNING OUT OF DISK SPACE?

Show me the command that you were running that caused the problem.
Also, what OS versions did this failure happen on?
[color=blue]
>Maybe it is relevant to the bug that the upper and lower
>directories are also quite big (0.8-1.5GB)[/color]

What is an "upper directory"?
[color=blue]
>I repeat that the bug is related to the 2GB limit since it appeared
>when I crossed it, around monday, when I updated
>the directory from the NASA site.[/color]

Kerry was nominated the Democratic candidate for President and a
lot of equipment I had connected to my phone line and network got
fried, so it must be his fault. My air conditioning repairman (the
A/C electronics got fried too) believes it had something to do with
lightning, though.
[color=blue]
>Sorry but I do not know if I can answer more, but it
>is VERY EASY TO REPRODUCE IT.
>
>Just make a program that makes random files of
>33-300K until you get 3GB, then do some directories and
>make files.[/color]

Did you check this filesystem for consistency before (and after,
if possible) trying this? (Linux: fsck. Windows: scandisk or
whatever they're calling it now).

I find it very easy to FAIL to reproduce it, on Linux and FreeBSD.

I doubt I'm going to try this on Windows. I can't figure out
how to install a version of Windows that won't get instantly infected
if I connect it to the net. (Don't say Windows Update unless you
know how to use it *WITHOUT* connecting to the net until *AFTER* the
update is complete.)

Gordon L. Burditt

[jim green]

Re: Signed and unsigned int

"jacob navia" <jacob@jacob.re mcomp.fr> writes:
[color=blue]
> It is when the contents of a directory exceed 2GB. Each
> of the files is between 30-300K.
>
> Just make a program that makes random files of
> 33-300K until you get 3GB, then do some directories and
> make files.[/color]

OK, Ive 3GB spare. My script is

# file size = 100k
$count = 200;
$bs = 512;

# 30,000 files is, then, 3GB
foreach $n (1...30000)
{
$file = sprintf "2gb/2gb-%.5i.dat",$n;
$cmd = "dd if=/dev/zero of=$file bs=$bs count=$count";
print "$cmd\n";
system($cmd);
}

(perl, of course). The directory now contains 30,000 of files

hardy:/data/other> ls -1 2gb | grep dat | wc -l
30000

they are all in the size range 30-300K

hardy:/data/other> ls -l 2gb | head
total 3120000
-rw-r--r-- 1 jjg users 102400 Aug 11 20:51 2gb-00001.dat
-rw-r--r-- 1 jjg users 102400 Aug 11 20:51 2gb-00002.dat
-rw-r--r-- 1 jjg users 102400 Aug 11 20:51 2gb-00003.dat
-rw-r--r-- 1 jjg users 102400 Aug 11 20:51 2gb-00004.dat

and the total size is > 2GB

hardy:/data/other> du -k 2gb
3120712 2gb

No problems with the file system etc. This is an ext3 filesystem
on a software raid partition on Linux 2.4.20

hardy:/data/other> df -k 2gb
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md0 115346344 105401392 4085608 97% /data

I dont think this is an generic problem

-j