Manipulating binary data

[brainflakes.org@googlemail.com]

Hi guys,

I need to manipulate binary data (8 bit) stored in a 2 dimensional
array.

I've tried various methods (arrays, using a string filled with chr(0),
using gd lib) and so far the fastest way I've found is to actually
create an 8-bit image in GD and use imagecolorat and imagesetpixel to
read and write the data.

Are there actually any proper binary extensions or is using gd lib the
way to go (as I guess it's just dealing with binary data as a 2d array
anyway)

Andrew

[Schraalhans Keukenmeester]

Re: Manipulating binary data

On Sun, 29 Apr 2007 09:36:38 -0700, brainflakes.org wrote:

Hi guys,
>
I need to manipulate binary data (8 bit) stored in a 2 dimensional
array.
>
I've tried various methods (arrays, using a string filled with chr(0),
using gd lib) and so far the fastest way I've found is to actually
create an 8-bit image in GD and use imagecolorat and imagesetpixel to
read and write the data.
>
Are there actually any proper binary extensions or is using gd lib the
way to go (as I guess it's just dealing with binary data as a 2d array
anyway)
>
Andrew

What kind if wizardry are you trying to achieve? Maybe a bit of your code
and an explanation of what you are after makes helping easier.

You are aware of the bitwise operators: (are you?)
$a & $b And Bits that are set in both $a and $b are set.
$a | $b Or Bits that are set in either $a or $b are set.
$a ^ $b Xor Bits that are set in $a or $b but not both are set.
~ $a Not Bits that are set in $a are not set, and vice versa.
$a << $b Shift left Shift the bits of $a $b steps to the left
(each step means "multiply by two")
$a >$b Shift right Shift the bits of $a $b steps to the right
(each step means "divide by two")


Sh.

[Chung Leong]

Re: Manipulating binary data

On Apr 29, 6:36 pm, brainflakes.... @googlemail.com wrote:

Hi guys,
>
I need to manipulate binary data (8 bit) stored in a 2 dimensional
array.
>
I've tried various methods (arrays, using a string filled with chr(0),
using gd lib) and so far the fastest way I've found is to actually
create an 8-bit image in GD and use imagecolorat and imagesetpixel to
read and write the data.
>
Are there actually any proper binary extensions or is using gd lib the
way to go (as I guess it's just dealing with binary data as a 2d array
anyway)
>
Andrew

I doubt that's the optimal solution. Function calls are relatively
expensive in PHP. The difference in speed you observed between the
different methods can probably be attributed to the number of calls
made per element.

Explain exactly what you're trying to do. There probably is a better
approach.

[brainflakes.org@googlemail.com]

Re: Manipulating binary data

I doubt that's the optimal solution. Function calls are relatively
expensive in PHP. The difference in speed you observed between the
different methods can probably be attributed to the number of calls
made per element.
>
Explain exactly what you're trying to do. There probably is a better
approach.

I'm creating some custom image processing filters on 8bit greyscale
images

for($x=0; $x<$width; $x++){
for($y=0; $x<$height; $y++){
check value of pixels (0-255) in a 10x10 area around given pixel
calculate new value
plot value at x,y (0-255)
}
}

I figured the best way to do it would be to just "allocate" a memory
space of $width * $height bytes and copy the image into that, which I
guess is done by creating a string of that length. However even just
plotting to $string[$n++]=chr($val) (no co-ord calculations) is slower
then imagesetpixel() on an 8-bit image.

For some reason tho I can't actually render directly from the 8-bit
image (I use imagesetcolor() to set colours in a greyscale gradient
but imagepng complains there's the wrong number of colours) so I have
to copy the image in and out of a truecolour image anyway.

I have done some further optimisations on the actual processing code
and improved the speed about 10-fold, but if there's a faster way to
read/write 8bit data I'm still interested as every little helps when
you're doing batch processing. (tho I guess I should probably be doing
it in C or .net anyway ;)

Andrew

[Toby A Inkster]

Re: Manipulating binary data

brainflakes.org wrote:

Are there actually any proper binary extensions or is using gd lib the
way to go (as I guess it's just dealing with binary data as a 2d array
anyway)

Assuming that each "row" of data is of the same length, and each "column"
has the same height (which I think is a fairly safe assumption if you're
modelling the data as a rectangular image) then, why not just store the
data as a string. That is, instead of storing a 3x3 array of data as this:

$array = array(
array(0x61, 0x62, 0x63),
array(0x31, 0x32, 0x33),
array(0x78, 0x79, 0x7A)
);

you store it as:

$string = 'abc123xyz';

Which is roughly how a 2-dimensional array is internally stored by a C
program.

Instead of this:

$cell = $array[$x][$y];

You use:

define(ROW_LENG TH, 3);
$cell = $string[(ROW_LENGTH*$x) + $y];

And instead of this:

$array[$x][$y] = 0x64;

this:

$string[(ROW_LENGTH*$x) + $y] = chr(0x64);


--
Toby A Inkster BSc (Hons) ARCS

http://tobyinkster.co.uk/

Geek of ~ HTML/SQL/Perl/PHP/Python*/Apache/Linux

* = I'm getting there!

[Chung Leong]

Re: Manipulating binary data

On Apr 30, 11:39 am, brainflakes.... @googlemail.com wrote:

I doubt that's the optimal solution. Function calls are relatively
expensive in PHP. The difference in speed you observed between the
different methods can probably be attributed to the number of calls
made per element.

>

Explain exactly what you're trying to do. There probably is a better
approach.

>
I'm creating some custom image processing filters on 8bit greyscale
images
>
for($x=0; $x<$width; $x++){
for($y=0; $x<$height; $y++){
check value of pixels (0-255) in a 10x10 area around given pixel
calculate new value
plot value at x,y (0-255)
}
>
}
>
I figured the best way to do it would be to just "allocate" a memory
space of $width * $height bytes and copy the image into that, which I
guess is done by creating a string of that length. However even just
plotting to $string[$n++]=chr($val) (no co-ord calculations) is slower
then imagesetpixel() on an 8-bit image.
>
For some reason tho I can't actually render directly from the 8-bit
image (I use imagesetcolor() to set colours in a greyscale gradient
but imagepng complains there's the wrong number of colours) so I have
to copy the image in and out of a truecolour image anyway.
>
I have done some further optimisations on the actual processing code
and improved the speed about 10-fold, but if there's a faster way to
read/write 8bit data I'm still interested as every little helps when
you're doing batch processing. (tho I guess I should probably be doing
it in C or .net anyway ;)
>
Andrew

Hmmm, that sort of processing will likely be slow in any event. Here
are some tips:

A hash look up is faster than calling chr() or ord(). So you can speed
things up by employing two tables:

$chr = array( 0 ="\x00", 1 ="\x01", 2 ="\x02" ..., 0xFF =>
"\xFF" );
$ord = array_flip($chr );


Unpacking the binary data into an array of ints with unpack() and then
repacking the result with pack() could be faster than repeated binary-
to-numeric conversions.


for() loops are slower than foreach(). Using the range() function, you
can create two arrays holding the indexes then go through them with
foreach() instead of incrementing them.


Loops have overhead. You can squeeze out some of that by unrolling
tight loops. What you would do is dynamically write out the PHP code
that'd execute by a full iteration of the loop, then create a lambda
function using create_function (). I used that technique in my
imagecreatefrom bmp() routine(http://www.conradish.net/bobo/
show_source.php ?filename=bmp.p hp).


When possible, use array_map() or array_walk() instead of looping.

[brainflakes.org@googlemail.com]

Re: Manipulating binary data

>
you store it as:
>
$string = 'abc123xyz';
>
Which is roughly how a 2-dimensional array is internally stored by a C
program.

As I said I've tried that, however using GD lib is actually 30-40%
faster then manipulating a string

"I figured the best way to do it would be to just "allocate" a memory
space of $width * $height bytes and copy the image into that, which I
guess is done by creating a string of that length. However even just
plotting to $string[$n++]=chr($val) (no co-ord calculations) is slower
then imagesetpixel() on an 8-bit image"

[Chung Leong]

Re: Manipulating binary data

On Apr 30, 11:49 am, Toby A Inkster <usenet200...@t obyinkster.co.u k>
wrote:

brainflakes.org wrote:

Are there actually any proper binary extensions or is using gd lib the
way to go (as I guess it's just dealing with binary data as a 2d array
anyway)

>
Assuming that each "row" of data is of the same length, and each "column"
has the same height (which I think is a fairly safe assumption if you're
modelling the data as a rectangular image) then, why not just store the
data as a string. That is, instead of storing a 3x3 array of data as this:
>
$array = array(
array(0x61, 0x62, 0x63),
array(0x31, 0x32, 0x33),
array(0x78, 0x79, 0x7A)
);
>
you store it as:
>
$string = 'abc123xyz';
>
Which is roughly how a 2-dimensional array is internally stored by a C
program.
>
Instead of this:
>
$cell = $array[$x][$y];
>
You use:
>
define(ROW_LENG TH, 3);
$cell = $string[(ROW_LENGTH*$x) + $y];
>
And instead of this:
>
$array[$x][$y] = 0x64;
>
this:
>
$string[(ROW_LENGTH*$x) + $y] = chr(0x64);
>
--
Toby A Inkster BSc (Hons) ARCShttp://tobyinkster.co. uk/
Geek of ~ HTML/SQL/Perl/PHP/Python*/Apache/Linux
>
* = I'm getting there!

Accessing a character within a string is going to be slower than
accessing an element in an array. It's true that PHP can very quickly
find the value given the index, but it has to create a new string
object to hold that character ($string[$i] is a one character string).
Memory allocation takes more time than a hash look-up, hence negative
saving.

The key to performance here is to reduce the amount of work in the
inner loop, as it's going to be run tens of thousands of times.
Converting the binary data to numbers and breaking them into an array
per line ahead of time will be likely be faster.

[Aerik]

Re: Manipulating binary data

I'd love to hear how you end up doing this, if you don't mind.

Aerik