Unicode strings, struct, and files

[Tom Plunket]

I am building a file with the help of the struct module.

I would like to be able to put Unicode strings into this file, but I'm
not sure how to do it.

The format I'm trying to write is basically this C structure:

struct MyFile
{
int magic;
int flags;
short otherFlags;
char pad[22];

wchar_t line1[32];
wchar_t line2[32];

// ... other data which is easy. :)
};

(I'm writing data on a PC to be read on a big-endian machine.)

So I can write the four leading members with the output of
struct.pack('>I IH22x', magic, flags, otherFlags). Unfortunately I
can't figure out how to write the unicode strings, since:

message = unicode('Hello, world')
myFile.write(me ssage)

results in 'message' being converted back to a string before being
written. Is the way to do this to do something hideous like this:

for c in message:
myFile.write(st ruct.pack('>H', ord(unicode(c)) ))

?

Thanks from a unicode n00b,
-tom!

[John Machin]

Re: Unicode strings, struct, and files


Tom Plunket wrote:

I am building a file with the help of the struct module.
>
I would like to be able to put Unicode strings into this file, but I'm
not sure how to do it.
>
The format I'm trying to write is basically this C structure:
>
struct MyFile
{
int magic;
int flags;
short otherFlags;
char pad[22];
>
wchar_t line1[32];
wchar_t line2[32];
>
// ... other data which is easy. :)
};
>
(I'm writing data on a PC to be read on a big-endian machine.)
>
So I can write the four leading members with the output of
struct.pack('>I IH22x', magic, flags, otherFlags). Unfortunately I
can't figure out how to write the unicode strings, since:
>
message = unicode('Hello, world')
myFile.write(me ssage)
>
results in 'message' being converted back to a string before being
written. Is the way to do this to do something hideous like this:
>
for c in message:
myFile.write(st ruct.pack('>H', ord(unicode(c)) ))
>
?

I'd suggest UTF-encoding it as a string, using the encoding that
matches whatever wchar means on the target machine, for example
assuming bigendian and sizeof(wchar) == 2:

utf_line1 = unicode_line1.e ncode('utf_16_b e')
etc
struct.pack(">. ........64s64s" , ......, utf_line1, utf_line2)
Presumes (1) you have already checked that you don't have more than 32
characters in each "line" (2) padding with unichr(0) is acceptable.

HTH,
John

[Tom Plunket]

Re: Unicode strings, struct, and files

John Machin wrote:

message = unicode('Hello, world')
myFile.write(me ssage)

results in 'message' being converted back to a string before being
written. Is the way to do this to do something hideous like this:

for c in message:
myFile.write(st ruct.pack('>H', ord(unicode(c)) ))

>
I'd suggest UTF-encoding it as a string, using the encoding that
matches whatever wchar means on the target machine, for example
assuming bigendian and sizeof(wchar) == 2:

Ahh, this is the info that my trawling through the documentation
didn't let me find!

Thanks a bunch.

utf_line1 = unicode_line1.e ncode('utf_16_b e')
etc
struct.pack(">. ........64s64s" , ......, utf_line1, utf_line2)
Presumes (1) you have already checked that you don't have more than 32
characters in each "line" (2) padding with unichr(0) is acceptable.

This works frighteningly well. ;)


-tom!