Faster way to write in a file

[LilacSkin]

Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

Tk.

[santosh]

Re: Faster way to write in a file

LilacSkin wrote:

Hi,
>
Currently, I write an signed long long in a file with the fprintf
function:
>
signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;
>
pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);
>
for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}
>
The problem is that function is very very slow !
>
In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

[LilacSkin]

Re: Faster way to write in a file

On 13 fév, 13:47, santosh <santosh....@gm ail.comwrote:

LilacSkin wrote:

Hi,

>

Currently, I write an signed long long in a file with the fprintf
function:

>

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

>

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

>

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

>

The problem is that function is very very slow !

>

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

>
size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

It's not working !

[santosh]

Re: Faster way to write in a file

LilacSkin wrote:

On 13 fév, 13:47, santosh <santosh....@gm ail.comwrote:

>LilacSkin wrote:

Hi,

>>

Currently, I write an signed long long in a file with the fprintf
function:

>>

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

>>

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

>>

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

>>

The problem is that function is very very slow !

>>

In several forums, I saw that fwrite function is better, but I
don't know how to use it.
Can you help me, please ?

>>
> size_t rc;
> rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
> if (rc != 1048576) {
> puts("Write error.");
> /* ... */
> }

>
It's not working !

How exactly? Did you check that the elements parameter is correct? What
is the value that rc has after the fwrite() call? Is pFilec pointing to
a valid stream for which you have appropriate permissions?

[Mark Bluemel]

Re: Faster way to write in a file

santosh wrote:

LilacSkin wrote:
>

>Hi,
>>
>Currently, I write an signed long long in a file with the fprintf
>function:
>>
>signed long long * pData = NULL;
>unsigned long long k = 0;
>unsigned long DataAvail = 0 ;
>>
>pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
>(65536*16*8) / 8);
>>
>for ( k = 0; k < DataAvail/8; k ++ ) {
> fprintf ( pFilec, "%lli\t", pData[k]);
>}
>>
>The problem is that function is very very slow !
>>
>In several forums, I saw that fwrite function is better, but I don't
>know how to use it.
>Can you help me, please ?

>
size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

[Morris Dovey]

Re: Faster way to write in a file

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp.lang.c/browse_thread/thread/eb3297730a3cc476/995f74496633fe3a?lnk=st&q=#995f74496633fe3a

(mind the wrap)

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA

http://www.iedu.com/DeSoto

[Mark Bluemel]

Re: Faster way to write in a file

Morris Dovey wrote:

Mark Bluemel wrote:
>

>How does dumping the data in binary form compare to writing a
>tab-delimited textual representation?

>
If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

My point was that the OP had clearly been trying to produce a textual
representation.

From context, I would not have expected the OP to understand why
Santosh's suggestion was not compatible with his/her original goal.

[Malcolm McLean]

Re: Faster way to write in a file

"LilacSkin" <lpaulo07@iseb. frwrote in message news

Hi,
>
Currently, I write an signed long long in a file with the fprintf
function:
>
signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;
>
pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);
>
for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}
>
The problem is that function is very very slow !
>
In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?
>

Replace %lli with %llx and measure the speedup.
Is hexadecimal output acceptable to you?

--
Free games and programming goodies.

http://www.personal.leeds.ac.uk/~bgy1mm

[LilacSkin]

Re: Faster way to write in a file

On 13 fév, 15:13, Morris Dovey <mrdo...@iedu.c omwrote:

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

>
If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.
>
A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:
>
http://groups.google.com/group/comp....d/thread/eb329...
>
(mind the wrap)
>
--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

Thanks !

[LilacSkin]

Re: Faster way to write in a file

On 13 fév, 19:51, LilacSkin <lpaul...@iseb. frwrote:

On 13 fév, 15:13, Morris Dovey <mrdo...@iedu.c omwrote:
>
>
>

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

>

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

>

A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

>

http://groups.google.com/group/comp....d/thread/eb329...

>

(mind the wrap)

>

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

>
Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?
>
Thanks !

In fact the most important thing is to write faster.
I have 1,6 Mbyte/s of raw data to write during a couple of days.
Because of the amount of data, I can't reformat their after.
The data are stored in a char buffer and I need to put in a file as a
signed long long.
The size of the buzzer is 8 Mbyte, refreshed every 5 sec.

I think the best way is to format them and use a fwrite or _write
function .

[Morris Dovey]

Re: Faster way to write in a file

LilacSkin wrote:

>
On 13 fév, 15:13, Morris Dovey <mrdo...@iedu.c omwrote:

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp....d/thread/eb329...

(mind the wrap)

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

>
Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

You probably won't want to use _that_ convert function because it
does more than you want (double -string) when you're really
wanting (long long -string).

Just throw away the part that deals with the fractional part of
the double - and if all of your values are positive, throw away
the sign logic, too. That won't leave very much - and it
shouldn't waste many cycles.

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA

http://www.iedu.com/DeSoto

[LilacSkin]

Re: Faster way to write in a file

#include <stdio.h>

void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;

if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;

*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));

while (p >= s) *p-- = '0';

if (sign) *s = '-';

}

int main(void)
{ long long test = -123456789012345 ;
char buffer[64];

convert(test,bu ffer,25,5);
printf("'%s'\n" ,buffer);
// then fwrite(buffer.. .)

return 0;

}

[Morris Dovey]

Re: Faster way to write in a file

LilacSkin wrote:

#include <stdio.h>
>
void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;
>
if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = '0';
if (sign) *s = '-';
}

Pretty close. Since you're not working with a double, you can
dispense with the fractional part logic and shorten to something
like:

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA

http://www.iedu.com/DeSoto

[Ben Bacarisse]

Re: Faster way to write in a file

LilacSkin <lpaulo07@iseb. frwrites:

#include <stdio.h>
>
void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;
>
if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;
>
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
>
while (p >= s) *p-- = '0';
>
if (sign) *s = '-';
>
}
>
int main(void)
{ long long test = -123456789012345 ;
char buffer[64];
>
convert(test,bu ffer,25,5);
printf("'%s'\n" ,buffer);
// then fwrite(buffer.. .)
>
return 0;
>
}

You may want to re-consider. Numbers are usually converted to a
decimal string representation so the we (humans) can read them. If
your program is producing such a volume of data that it is hard to get
converted output fast enough, will the resulting data ever be read by
a person? I doubt it. You'd need a program to scan just one second's
worth.

It might pay to store the data as native binary numbers. That is
probably what was being suggested when someone said use "fwrite".

Of course, if the output *has* to be processed by an existing program
that needs decimal input, then the conversion has to happen some time,
but it could be done later, at leisure, so to speak. It might even be
possible to process into decimal only the portion you are interested in.
Maybe the binary data could be indexed for rapid access to specific
parts. Without knowing the ultimate fate of this data, it is hard to
be more specific, but I suggest you look beyond your current headache
of not being able to use fprintf because it seems too slow.

--
Ben.

[Mark Bluemel]

Re: Faster way to write in a file

LilacSkin wrote:

I have 1,6 Mbyte/s of raw data to write during a couple of days.
Because of the amount of data, I can't reformat their after.
The data are stored in a char buffer and I need to put in a file as a
signed long long.

So why were you even trying to use printf, which is for formatted
output?

If you have to write to the file as signed long long, then fwrite is
appropriate.

You really could help us to help you if you explained the requirement
better.