Byte swapping help please

[Ann]

I am opening a file which looks like 0xABCDEF01 on another machine but
0x01EFCDAB on my machine.

Is this a byte swapping?

Could anyone give a good way to check if bytes are being swapped? (code
should work smoothly across different machine.)

Thanks,
Ann

[Craig Ruff]

Re: Byte swapping help please

In article <1144954044.828 711.279960@z34g 2000cwc.googleg roups.com>,
Ann <ps_kal@yahoo.c om> wrote:[color=blue]
>I am opening a file which looks like 0xABCDEF01 on another machine but
>0x01EFCDAB on my machine.
>
>Is this a byte swapping?[/color]

Or possbily spouse swapping.
[color=blue]
>Could anyone give a good way to check if bytes are being swapped? (code
>should work smoothly across different machine.)[/color]

Perhaps we should just send the code directly to your instructor so
we can get the credit for your homework?
--

Craig Ruff NCAR cruff@ucar.edu
303-497-1211 P.O. Box 3000
Boulder, CO 80307

[Keith Thompson]

Re: Byte swapping help please

"Ann" <ps_kal@yahoo.c om> writes:[color=blue]
> I am opening a file which looks like 0xABCDEF01 on another machine but
> 0x01EFCDAB on my machine.
>
> Is this a byte swapping?[/color]

Looks like it.
[color=blue]
> Could anyone give a good way to check if bytes are being swapped? (code
> should work smoothly across different machine.)[/color]

In principle, there is no reliable way to tell. If you read a 32-bit
unsigned integer from a file and get a value of 0xABCDEF01, how can
you know whether it should be 0xABCDEF01 or 0x01EFCDAB? Without more
information, you can't. Even with more information, you may not be
able to tell.

If you're storing binary data in a file, byte ordering is only one of
the problems you can run into. Sizes of types can vary across
different implementations ; so cah floating-point representations .

The safest approach is to write *only* byte data. For example, if you
want to write an integer value 0x01EFCDAB to a file, you can read and
write the individual bytes (0x01, 0xEF, 0xCD, 0xAB) in a fixed order.
Or you can write a textual representation of the number, which also
has the advantage of letting you view the file with a text editor.

Strictly speaking, you might still have problems on systems with byte
sizes bigger than 8 bits, or with non-ASCII character sets; the former
is unlikely to arise in practice, and the latter can be solved with
textual conversion tools. (There are systems, mostly DSPs, with bytes
bigger than 8 bits, but they're embedded systems, and you're not
likely to need to share files with them.)

If you must write raw binary data to a file, you might add information
to the file header indicating how the data is formatted.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Keith Thompson]

Re: Byte swapping help please

cruff@ucar.edu (Craig Ruff) writes:[color=blue]
> In article <1144954044.828 711.279960@z34g 2000cwc.googleg roups.com>,
> Ann <ps_kal@yahoo.c om> wrote:[color=green]
>>I am opening a file which looks like 0xABCDEF01 on another machine but
>>0x01EFCDAB on my machine.
>>
>>Is this a byte swapping?[/color]
>
> Or possbily spouse swapping.
>[color=green]
>>Could anyone give a good way to check if bytes are being swapped? (code
>>should work smoothly across different machine.)[/color]
>
> Perhaps we should just send the code directly to your instructor so
> we can get the credit for your homework?[/color]

I didn't see any indication that this was homework. (If it was, it's
a very poorly stated problem; as I mentioned elsethread, there is no
reliable way to detect the byte ordering of a binary file.)

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Roberto Waltman]

Re: Byte swapping help please

Keith Thompson wrote:
[color=blue]
>... as I mentioned elsethread, there is no
>reliable way to detect the byte ordering of a binary file.[/color]

I believe there is, if you are allowed to "cheat" by using an
auxiliary file, or a field with a known value at the beginning of the
file.
Assuming, for example, that the file will be used to store 32 bit
integers, writing 0x12345678 as the first 4 octets will provide you
will all the information you need to decode the following data
correctly.

[ray]

Re: Byte swapping help please

On Thu, 13 Apr 2006 11:47:24 -0700, Ann wrote:
[color=blue]
> I am opening a file which looks like 0xABCDEF01 on another machine but
> 0x01EFCDAB on my machine.
>
> Is this a byte swapping?
>
> Could anyone give a good way to check if bytes are being swapped? (code
> should work smoothly across different machine.)
>
> Thanks,
> Ann[/color]

One is 'big-endian' and one is 'little-endian'. That is exactly what htonl
and ntohl are for (host to network long and network to host long).

[Al Balmer]

Re: Byte swapping help please

On Thu, 13 Apr 2006 15:57:40 -0400, Roberto Waltman
<usenet@rwaltma n.net> wrote:
[color=blue]
>Keith Thompson wrote:
>[color=green]
>>... as I mentioned elsethread, there is no
>>reliable way to detect the byte ordering of a binary file.[/color]
>
>I believe there is, if you are allowed to "cheat" by using an
>auxiliary file, or a field with a known value at the beginning of the
>file.
>Assuming, for example, that the file will be used to store 32 bit
>integers, writing 0x12345678 as the first 4 octets will provide you
>will all the information you need to decode the following data
>correctly.[/color]

That can be a useful technique. William Waite used something like that
to distribute the Stage 2 macro processor. As I recall, the Fortran
bootstrap read a record containing the character set to be used.

--
Al Balmer
Sun City, AZ

[Roberto Waltman]

Re: Byte swapping help please

Al Balmer wrote:

<OT> ( I think..)[color=blue]
> William Waite used something like that
>to distribute the Stage 2 macro processor. As I recall, the Fortran
>bootstrap read a record containing the character set to be used.[/color]

What is/was "the Stage 2 macro processor" ?
</OT>

[Al Balmer]

Re: Byte swapping help please

On Thu, 13 Apr 2006 16:49:16 -0400, Roberto Waltman
<usenet@rwaltma n.net> wrote:
[color=blue]
>Al Balmer wrote:
>
><OT> ( I think..)[/color]

OT in comp.lang.c. I plead ignorance about the other two groups <g>.
[color=blue][color=green]
>> William Waite used something like that
>>to distribute the Stage 2 macro processor. As I recall, the Fortran
>>bootstrap read a record containing the character set to be used.[/color]
>
>What is/was "the Stage 2 macro processor" ?[/color]

A rather nice macro processor designed to be ported to any system
which had a Fortran compiler. Years ago, I used it to implement a
system called "SAP" (Structured Assembler Programming) which was used
successfully to implement a number of process control products. Here's
an abstract of an early paper:

http://hopl.murdoch.edu.au/showlanguage2.prx?exp=534

########
* Waite, W. M. "The Mobile Programming System: STAGE2" view
details Abstract: STAGE2 is the second level of a bootstrap sequence
which is easily implemented on any computer. It is a flexible,
powerful macro processor designed specifically as a tool for
constructing machine-independent software. In this paper the features
provided by STAGE2 are summarized, and the implementation techniques
which have made it possible to have STAGE2 running on a new machine
with less than one man-week of effort are discussed. The approach has
been successful on over 15 machines of widely varying characteristics .
DOI
in [ACM] CACM 13(09) (Sep 1970) view details
########

The published papers were not quite sufficient to implement the
system. For that, the best resource was the book:

Waite, W. M. Implementing Software for Non-numeric Applications, P-H
1973
[color=blue]
></OT>[/color]

--
Al Balmer
Sun City, AZ

[Roberto Waltman]

Re: Byte swapping help please

Al Balmer wrote:
[color=blue][color=green]
>>What is/was "the Stage 2 macro processor" ?[/color]
>
>A rather nice macro processor designed to be ported to any system
>which had a Fortran compiler. Years ago, I used it to implement a
>system called "SAP" (Structured Assembler Programming) which was used
>successfully to implement a number of process control products. Here's
>an abstract of an early paper:
>http://hopl.murdoch.edu.au/showlanguage2.prx?exp=534[/color]

Thanks for the info, looks interesting. Of course now I must learn
what FLUB and LIMP are, and then... There goes my weekend...

[danielhe99@gmail.com]

Re: Byte swapping help please

Yes, the id stored at the beginning of the binary file is called
"magic number".

Here is an example:

http://aslan.smnd.sk/anino/programming/gettext-doc/gettext_6.html

[Keith Thompson]

Re: Byte swapping help please

danielhe99@gmai l.com writes:[color=blue]
> Yes, the id stored at the beginning of the binary file is called
> "magic number".
>
> Here is an example:
> http://aslan.smnd.sk/anino/programmi...gettext_6.html[/color]

Please read <http://cfaj.freeshell. org/google/>.

Yes, if you store the right information in the file, it's possible to
determine its endianness. My point was that there's no *general* way
to do this. If I use fwrite() to write, say, an array of integers to
a binary file, there's no way to determine the endianness of the file
unless it's indicated explicitly, or unless I know something about
the expected values.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Andy Glew]

Re: Byte swapping help please

Roberto Waltman <usenet@rwaltma n.net> writes:
[color=blue]
> Al Balmer wrote:
>[color=green][color=darkred]
> >>What is/was "the Stage 2 macro processor" ?[/color]
> >
> >A rather nice macro processor designed to be ported to any system
> >which had a Fortran compiler. Years ago, I used it to implement a
> >system called "SAP" (Structured Assembler Programming) which was used
> >successfully to implement a number of process control products. Here's
> >an abstract of an early paper:
> >http://hopl.murdoch.edu.au/showlanguage2.prx?exp=534[/color]
>
> Thanks for the info, looks interesting. Of course now I must learn
> what FLUB and LIMP are, and then... There goes my weekend...[/color]


Cool.

Circa 2000 I defined (and had somebody implement) a macro language
that had a property similar to something I saw on a quick scan of
Stage2: the text after expansion was completely reprocessed by all
remaining patterns, repeatedly.

Now, sure, macro languages like CPP will expand a macro, then will
expand all macros in the macro, etc. But, to the best of my
knowledge, they have a single pattern match going on - macro
invocation such as FOO().

The fun part was
a) defining "best match" in a way that users found meaningful
b) reparsing completely - e.g. Foo##Bar() might concatenate,
and then expand FooBar()
c) defining things in a way so that phase ordering did not
produce unpleasant artifacts.

Unfortunately, that project evaporated when I left Intel.

[Charles Allen]

Re: Byte swapping help please

Ann:[color=blue][color=green]
>> Could anyone give a good way to check if bytes are being swapped? (code
>> should work smoothly across different machine.)[/color][/color]

Keith Thompson:[color=blue]
> If you're storing binary data in a file, byte ordering is only one of
> the problems you can run into. Sizes of types can vary across
> ...
> The safest approach is to write *only* byte data. For example, if you[/color]

Depending on what level you control the I/O and how often this is
going to occur, you could also look at XDR, or something higher level
like netCDF.

--
Charles Allen

[Andrew Reilly]

Re: Byte swapping help please

On Fri, 14 Apr 2006 02:10:06 +0000, Keith Thompson wrote:[color=blue]
> Yes, if you store the right information in the file, it's possible to
> determine its endianness. My point was that there's no *general* way
> to do this. If I use fwrite() to write, say, an array of integers to
> a binary file, there's no way to determine the endianness of the file
> unless it's indicated explicitly, or unless I know something about
> the expected values.[/color]

If the numbers represent a white, random sequence, then it might not
matter which order you read them. Maybe that's good enough for the OP?

Personally, I favor the "write the bytes in the order you want them"
school. Even htonl and friends have the problem that you have to cast the
result to an unsigned char array, which can annoy the DSP processors that
you spoke of, earlier. They'll usually be quite happy with the arithmetic
of the explicit byte extraction approach, and if you have an octet-wide
peripheral or memory to stuff the results, you'll even get the same file...
(Luckily for the sanity of programmers, byte-addressability seems to be
becoming more popular in DSPs too, at least those that have some
expectation of being spoken to in C, some of the time.)

Cheers,

--
Andrew