xor incongruences

[Michele]

Hi,
I write a simple encoder in python and Java; they do the same
computations, with the same inputs: however they won't produce the same
output.
Let me explain with code.

First of all, you need a test file for input:
$ dd if=/dev/urandom of=test.img bs=1048576 count=1

I have attached the code.
As you can see we have xor-red the same inputs (bitlists are the same,
thus the selected blocks to xor are the same - you can easily see it,
whenever a block is xor-red both programs will print out its hash).
But here comes the strange: the random_block that the
create_random_b lock function returns is not the same: in Java this block
has an hash which is different from the Python one.
Why?

Thank you

import os
import sha
import sys

class EncoderDecoder( object):
def create_random_b lock(self,piece ,blocksize=1638 4):
if len(piece) % blocksize != 0:
raise Exception('size error')
self.N = len(piece)/blocksize
random_block = ['0']*blocksize
bitlist = [1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1]
for i in range(len(bitli st)):
if bitlist[i] == 1:
block = piece[i*blocksize:i*b locksize+blocks ize]
print '%d-%d %s' %(i*blocksize,i *blocksize+bloc ksize,sha.new(b lock).hexdigest ())
for j in range(blocksize ):
random_block[j] = chr(ord(random_ block[j]) ^ ord(block[j]))
print sha.new(''.join (random_block)) .hexdigest()
return ''.join(random_ block)

if __name__ == '__main__':
data = open('test.img' ,'rb').read()
x = EncoderDecoder( )
x.create_random _block(data)
sys.exit(0)

import java.io.FileInp utStream;
import java.io.IOExcep tion;
import java.io.Unsuppo rtedEncodingExc eption;
import java.security.M essageDigest;
import java.security.N oSuchAlgorithmE xception;

public class Test {
int N;

public byte[] create_random_b lock(byte[] piece)
throws NoSuchAlgorithm Exception, UnsupportedEnco dingException {
N = piece.length / 16384;
byte[] random_block = new byte[16384];
int[] bitlist = { 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0,
0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0,
1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1,
0, 1, 0, 1 };
for (int i = 0; i < N; i++) {
if (bitlist[i] == 1) {
byte[] block = new byte[16384];
for (int j = 0; j < block.length; j++) {
block[j] = piece[i * 16384 + j];
}
System.out.prin tln(i * 16384 + "-" + (i * 16384 + 16384)
+ " " + AeSimpleSHA1.SH A1(block));
for (int j = 0; j < random_block.le ngth; j++) {
random_block[j] = (byte) (random_block[j] ^ block[j]);
}
}
}
System.out.prin tln(AeSimpleSHA 1.SHA1(random_b lock));
return random_block;
}

public static void main(String[] args) throws IOException,
NoSuchAlgorithm Exception {
byte data[] = new byte[1024 * 1024];
FileInputStream fi = new FileInputStream ("test.img") ;
fi.read(data);
Test x = new Test();
x.create_random _block(data);
System.exit(0);

}

public static class AeSimpleSHA1 {
private static String convertToHex(by te[] data) {
StringBuffer buf = new StringBuffer();
for (int i = 0; i < data.length; i++) {
int halfbyte = (data[i] >>4) & 0x0F;
int two_halfs = 0;
do {
if ((0 <= halfbyte) && (halfbyte <= 9))
buf.append((cha r) ('0' + halfbyte));
else
buf.append((cha r) ('a' + (halfbyte - 10)));
halfbyte = data[i] & 0x0F;
} while (two_halfs++ < 1);
}
return buf.toString();
}

public static String SHA1(byte[] text) throws NoSuchAlgorithm Exception,
UnsupportedEnco dingException {
MessageDigest md;
md = MessageDigest.g etInstance("SHA-1");
byte[] sha1hash = new byte[40];
md.update(text) ;
sha1hash = md.digest();
return convertToHex(sh a1hash);
}
}

}

[bearophileHUGS@lycos.com]

Re: xor incongruences

Michele:

in Java this block has an hash which is different from the Python one.

Note that integer numbers in Python are multiprecision by default,
this may cause differences.

You can put some prints in various stages of the data flow (or
breakpoints for your debuggers, etc) to spot where the variable
contents start to differ.

Bye,
bearophile

[Peter Otten]

Re: xor incongruences

Michele wrote:

Hi,
I write a simple encoder in python and Java; they do the same
computations, with the same inputs: however they won't produce the same
output.
Let me explain with code.
>
First of all, you need a test file for input:
$ dd if=/dev/urandom of=test.img bs=1048576 count=1
>
I have attached the code.
As you can see we have xor-red the same inputs (bitlists are the same,
thus the selected blocks to xor are the same - you can easily see it,
whenever a block is xor-red both programs will print out its hash).
But here comes the strange: the random_block that the
create_random_b lock function returns is not the same: in Java this block
has an hash which is different from the Python one.
Why?

random_block = ['0']*blocksize

should be

random_block = ['\0']*blocksize

As John Machin already told you in another thread -- the character "0" is
not the same as the 0-byte "\0".

Peter

[John Machin]

Re: xor incongruences

On Oct 17, 7:02 am, Michele <mich...@nectar ine.itwrote:

Hi,
I write a simple encoder in python and Java; they do the same
computations, with the same inputs

No they don't.

however they won't produce the same
output.
Let me explain with code.

You have a strange understanding of the word "explain".

>
First of all, you need a test file for input:
$ dd if=/dev/urandom of=test.img bs=1048576 count=1
>
I have attached the code.
As you can see we have xor-red the same inputs (bitlists are the same,
thus the selected blocks to xor are the same - you can easily see it,
whenever a block is xor-red both programs will print out its hash).
But here comes the strange: the random_block that the
create_random_b lock function returns is not the same: in Java this block
has an hash which is different from the Python one.
Why?
>
Thank you
>
[test.py1K ]
        random_block = ['0']*blocksize

This initialises each element to '0'. ord('0') is 48, not 0. Either
I'm hallucinating, or I pointed this out to you in response to your
previous posting (within the last few days).

                    random_block[j] = chr(ord(random_ block[j]) ^ ord(block[j]))
>
[Test.java2K ]

                byte[] random_block = new byte[16384];

Presumably java initialises each element to 0 (or maybe random
gibberish); it is highly unlikely to be 48!

                                        random_block[j] = (byte) (random_block[j] ^ block[j]);