How to write array of doubles to stream without using a loop?

[John Dumais]

Hello,

I have been trying to figure out how to write an array of doubles (in
this specific case) to a binary stream without using a loop. What I
have been doing is...

foreach(double d in TraceData)
{
instanceOfBinar yWriter.Write(d );
}

The loop is introducing overhead I don't want. In unmanaged c++, I
would just do something like...

fwrite(buffer, sizeof(double), numArrayElement s, fileStream)

Is their an equivalent using c#?

Thanks,

[Sean Hederman]

Re: How to write array of doubles to stream without using a loop?

Surely fwrite would have a loop inside it? If it does, then surely it
doesn't matter if the loop occurs in your code or in library code?

"John Dumais" <nospam_john_du mais@agilent.co m> wrote in message
news:1105639468 .28323@cswreg.c os.agilent.com. ..[color=blue]
> Hello,
>
> I have been trying to figure out how to write an array of doubles (in this
> specific case) to a binary stream without using a loop. What I have been
> doing is...
>
> foreach(double d in TraceData)
> {
> instanceOfBinar yWriter.Write(d );
> }
>
> The loop is introducing overhead I don't want. In unmanaged c++, I would
> just do something like...
>
> fwrite(buffer, sizeof(double), numArrayElement s, fileStream)
>
> Is their an equivalent using c#?
>
> Thanks,[/color]

[Nicholas Paldino [.NET/C# MVP]]

Re: How to write array of doubles to stream without using a loop?

John,

You won't really be able to do that. It would be easier if you had
direct memory access, but unfortunately, you don't. The only thing that you
could do is do a conversion to a byte array, but you know that will
introduce overhead as well (that routine has to loop through as well to do
the conversion).

The only other option you would have is if your method can use unsafe
code. That way, you could cast the double array to a double pointer, and
then cast that to a pointer of a byte, and then pass the byte to the method
without having to loop through to get a byte array.

Hope this helps.

--
- Nicholas Paldino [.NET/C# MVP]
- mvp@spam.guard. caspershouse.co m

"John Dumais" <nospam_john_du mais@agilent.co m> wrote in message
news:1105639468 .28323@cswreg.c os.agilent.com. ..[color=blue]
> Hello,
>
> I have been trying to figure out how to write an array of doubles (in this
> specific case) to a binary stream without using a loop. What I have been
> doing is...
>
> foreach(double d in TraceData)
> {
> instanceOfBinar yWriter.Write(d );
> }
>
> The loop is introducing overhead I don't want. In unmanaged c++, I would
> just do something like...
>
> fwrite(buffer, sizeof(double), numArrayElement s, fileStream)
>
> Is their an equivalent using c#?
>
> Thanks,[/color]

[AMoose]

Re: How to write array of doubles to stream without using a loop?

> You won't really be able to do that. It would be easier if you had[color=blue]
> direct memory access, but unfortunately, you don't. The only thing that[/color]
you[color=blue]
> could do is do a conversion to a byte array, but you know that will
> introduce overhead as well (that routine has to loop through as well to do
> the conversion).[/color]

Take a look at Buffer.BlockCop y. It only works with primitive types. But I
remember looking at the rotor source code and its an internal call and does
a memcopy , memove, mem something but not a for loop. The docs say "This
class provides better performance for manipulating primitive types than
similar methods in the System.Array class.". Here is a sample that did work.

Dim Ds() As Integer = {1, 2, 3}
Dim I() As Byte = Array.CreateIns tance(GetType(B yte), Buffer.ByteLeng th(Ds))
Buffer.BlockCop y(Ds, 0, I, 0, Buffer.ByteLeng th(Ds))



"Nicholas Paldino [.NET/C# MVP]" <mvp@spam.guard .caspershouse.c om> wrote in
message news:ewrrRza%23 EHA.2452@TK2MSF TNGP14.phx.gbl. ..[color=blue]
> John,
>
> You won't really be able to do that. It would be easier if you had
> direct memory access, but unfortunately, you don't. The only thing that[/color]
you[color=blue]
> could do is do a conversion to a byte array, but you know that will
> introduce overhead as well (that routine has to loop through as well to do
> the conversion).
>
> The only other option you would have is if your method can use unsafe
> code. That way, you could cast the double array to a double pointer, and
> then cast that to a pointer of a byte, and then pass the byte to the[/color]
method[color=blue]
> without having to loop through to get a byte array.
>
> Hope this helps.
>
> --
> - Nicholas Paldino [.NET/C# MVP]
> - mvp@spam.guard. caspershouse.co m
>
> "John Dumais" <nospam_john_du mais@agilent.co m> wrote in message
> news:1105639468 .28323@cswreg.c os.agilent.com. ..[color=green]
> > Hello,
> >
> > I have been trying to figure out how to write an array of doubles (in[/color][/color]
this[color=blue][color=green]
> > specific case) to a binary stream without using a loop. What I have[/color][/color]
been[color=blue][color=green]
> > doing is...
> >
> > foreach(double d in TraceData)
> > {
> > instanceOfBinar yWriter.Write(d );
> > }
> >
> > The loop is introducing overhead I don't want. In unmanaged c++, I[/color][/color]
would[color=blue][color=green]
> > just do something like...
> >
> > fwrite(buffer, sizeof(double), numArrayElement s, fileStream)
> >
> > Is their an equivalent using c#?
> >
> > Thanks,[/color]
>
>[/color]

[Jon Skeet [C# MVP]]

Re: How to write array of doubles to stream without using a loop?

John Dumais <nospam_john_du mais@agilent.co m> wrote:[color=blue]
> I have been trying to figure out how to write an array of doubles (in
> this specific case) to a binary stream without using a loop. What I
> have been doing is...
>
> foreach(double d in TraceData)
> {
> instanceOfBinar yWriter.Write(d );
> }
>
> The loop is introducing overhead I don't want.[/color]

Have you measured the overhead? Is it definitely causing a problem?
There's a tendency to assume that things like this are bottlenecks
without experimentation - that may not be the case here, but it's worth
knowing before we get into much riskier code.

--
Jon Skeet - <skeet@pobox.co m>

http://www.pobox.com/~skeet

If replying to the group, please do not mail me too

[John Dumais]

Re: How to write array of doubles to stream without using a loop?

Jon Skeet [C# MVP] wrote:
[color=blue]
> Have you measured the overhead? Is it definitely causing a problem?
> There's a tendency to assume that things like this are bottlenecks
> without experimentation - that may not be the case here, but it's worth
> knowing before we get into much riskier code.
>[/color]

Not specifically in C#, because I can't get an apples-to-apples
comparison, but my experience has been that most run-time
libraries work better when you supply them with one hunk of data
as opposed to supplying many little hunks in a loop. The attached
example will illustrate. It's in C++, because I can't come up
with a reasonable comarison in C#. Sorry in advance -- I wrote
this illustration on my Linux machine. I don't have a Windows
PC at home. I ran the program 10 times in a loop and got the
following results.
[color=blue]
> Using loop
> Elapsed time: 2 seconds, 223613 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 725197 micro-seconds
> Using loop
> Elapsed time: 3 seconds, 256561 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 277398 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 224807 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 45266 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 215541 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 231558 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 221532 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 278577 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 220584 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 11936 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 215263 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 28349 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 223527 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 273473 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 226312 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 7940 micro-seconds
> Using loop
> Elapsed time: 2 seconds, 217676 micro-seconds
> Writing in a single hunk
> Elapsed time: 1 seconds, 23728 micro-seconds
>[/color]

The program...
[color=blue]
> #include <cstdio>
> #include <cstdlib>
> #include <sys/time.h>
>
> typedef void(*WriterFun cPtr)(FILE*, double*, size_t);
>
> void timeIt(WriterFu ncPtr funcPtr, FILE *fp, double *data,
> size_t numDataPoints)
> {
> timeval startTime = {0, 0};
> timeval endTime = {0, 0};
> timeval elapsedTime = {0, 0};
>
> if( ! gettimeofday(&s tartTime, 0)){
> funcPtr(fp, data, numDataPoints);
>
> if( ! gettimeofday(&e ndTime, 0)){
> if(startTime.tv _usec > endTime.tv_usec ){
> endTime.tv_usec += 1000000;
> endTime.tv_sec--;
> }
>
> elapsedTime.tv_ usec = endTime.tv_usec - startTime.tv_us ec;
> elapsedTime.tv_ sec = endTime.tv_sec - startTime.tv_se c;
>
> printf("Elapsed time: %ld seconds, %ld micro-seconds\n",
> elapsedTime.tv_ sec, elapsedTime.tv_ usec);
> }
> }
>
> }
>
> void checkFileConten ts(FILE *fp)
> {
> double data = 0;
>
> size_t numRead = 0;
>
> do{
> numRead = fread(&data, sizeof(data), 1, fp);
> printf("%lf\n", data);
> } while(numRead > 0);
> }
>
> void writeDataUsingL oop(FILE *fp, double *data, size_t numDataPoints)
> {
> for(size_t i = 0; i < numDataPoints; ++i){
> (void)fwrite(da ta++, sizeof(double), 1, fp);
> }
> }
>
> void writeDataInOneH unk(FILE *fp, double *data, size_t numDataPoints)
> {
> (void)fwrite(da ta, sizeof(double), numDataPoints, fp);
> }
>
> int main(void)
> {
> FILE *fp = fopen("data", "w");
> if(fp){
> const size_t numDataPoints = 8 * 1024 * 1024;
> double *data = (double*)malloc (numDataPoints * sizeof(double)) ;
> if(data){
> for(size_t i = 0; i < numDataPoints; ++i){
> data[i] = i;
> }
>
> printf("Using loop\n");
> timeIt(writeDat aUsingLoop, fp, data, numDataPoints);
>
> fclose(fp);
>
> fp = fopen("data", "w");
> if(fp){
> printf("Writing in a single hunk\n");
> timeIt(writeDat aInOneHunk, fp, data, numDataPoints);
>
> fclose(fp);
> fp = 0;
> }
>
> free(data);
> }
> }
>
> /*
> fp = fopen("data", "r");
> if(fp){
> checkFileConten ts(fp);
>
> fclose(fp);
> fp = 0;
> }
> */
>
> return 0;
> }
>[/color]

Thanks,