Mail extraction problem (something's wrong with split methods)

**Diez B. Roggisch** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

> File "OTPAenc_dec.py ", line 258, in decr[color=blue]
> enc = [int(elem) for elem in split]
> ValueError: invalid literal for int(): 6730'[/color]

The problem is the trailing ' in your number - that of course can't be
converted. And I see that the number 6573 has similar problems - it has a
leading '.

So your splitting code does not work, or your data is malformed - without
more information, I can't say anything about that, but it seems to me the
latter is the case.

--
Regards,

Diez B. Roggisch

**Luka Milkovic** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

On Sat, 11 Sep 2004 19:01:35 +0200, Diez B. Roggisch wrote:

[color=blue]
> The problem is the trailing ' in your number - that of course can't be
> converted. And I see that the number 6573 has similar problems - it has a
> leading '.[/color]

Yes, I know that, but I don't understand why it works normally for lists
under 350 bytes? It works perfectly...
[color=blue]
> So your splitting code does not work, or your data is malformed -
> without more information, I can't say anything about that, but it seems
> to me the latter is the case.[/color]

Data is actually not malformed, because before splitting it looks normal
(I mean, no ' or double quotes or other strange characters). The splitting
code is the problem, and I don't know how to fix it. I mean, if it would
be wrong, the smaller lists wouldn't work either, but it seems the
problems occur with big lists.

**Diez B. Roggisch** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

> Yes, I know that, but I don't understand why it works normally for lists[color=blue]
> under 350 bytes? It works perfectly...[/color]

That certainly has _nothing_ to do with the size of 350 - this snippet works
perfect:

len(",".join([str(i) for i in xrange(20000)]).split(','))

[color=blue][color=green]
>> So your splitting code does not work, or your data is malformed -
>> without more information, I can't say anything about that, but it seems
>> to me the latter is the case.[/color]
>
> Data is actually not malformed, because before splitting it looks normal
> (I mean, no ' or double quotes or other strange characters). The splitting
> code is the problem, and I don't know how to fix it. I mean, if it would
> be wrong, the smaller lists wouldn't work either, but it seems the
> problems occur with big lists.[/color]

As I proved above, it has nothing to do with that. Unless you provide actual
data I can't say more. I can only guess that 350 bytes has something to do
with line-boundaries or similar stuff - you hit some sort of special case
you didn't thing of or such a thing.

Do post the data, and I'm sure things will be sorted out soon.

--
Regards,

Diez B. Roggisch

**Peter Otten** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

Luka Milkovic wrote:
[color=blue]
> temp = [mail.read()]
> enc_txt = "\n".join(t emp)[/color]

These two lines an be simplified to

enc_text = mail.read()
[color=blue]
> begin = enc_txt.find(", '[")+len(", '[")
> ending = enc_txt.find("]', ")[/color]

A guess: you are not checking whether the ending "]', " is really found. It
may well be that it is not and therefore ending set to -1 -- which means
end_txt2 will contain all characters from 'begin' until the end of the
string excluding only the last character.
[color=blue]
> enc_txt2 = (enc_txt[begin:ending])
> mail.close()[/color]

next two lines are superfluous:
[color=blue]
> lines = enc_txt2.splitl ines()
> enc_txt3 = ' '.join([line.strip() for line in lines])[/color]
[color=blue]
> split = re.split(",", enc_txt3)[/color]

better IMHO:

split = enc_txt2.split( ",")

you need only one of the following lines:
[color=blue]
> enc = [int(elem) for elem in split]
> enc = map(int, split)[/color]

If my guess doesn't pinpoint the problem, I suggest that you post the actual
code and data that reproduces the error.

Peter

**Luka Milkovic** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

On Sat, 11 Sep 2004 20:11:42 +0200, Diez B. Roggisch wrote:
[color=blue]
> That certainly has _nothing_ to do with the size of 350 - this snippet
> works perfect:
>
> len(",".join([str(i) for i in xrange(20000)]).split(','))
>[/color]

The thing you said here led me to a tiny clue. The splitting code is
actually not the source of the problem: i made a list looking just like
ordinary e-mail many bytes long and it did it's job great.
The clue I was thinking about is e-mail format. Is there a possibility
that mails get specially formated after a particular size? Because, my
splitting code works great, but the source of the problem is obviously
e-mail, I checked temporary file into which e-mail is saved _before_
spliting and it too contains this strange quotes.

My mailing code looks like this:

def prompt(prompt):
return raw_input(promp t).strip()
posiljatelj = prompt("From: ")
primatelj = prompt("To: ").split()
host = prompt("SMTP host: ")
subject = "OTP"
msgin = priv
x = "\n---Code block---\n"
msg = ("From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n"
% (posiljatelj, string.join(pri matelj, ", "), subject))
msg = msg + x + msgin + x
print "Vas e-mail: ", msg
print ""
print "Duljina vaseg maila je " + `len(msg)` + " bajtova"
server = smtplib.SMTP(ho st)
server.set_debu glevel(1)
server.sendmail (posiljatelj, primatelj, msg)
server.quit()

I don't think there is problem in my actual SMTP code, it's more likely
that the problem lies inside the protocol itself or some special formating
style, I don't know.
[color=blue]
> As I proved above, it has nothing to do with that. Unless you provide
> actual data I can't say more. I can only guess that 350 bytes has
> something to do with line-boundaries or similar stuff - you hit some
> sort of special case you didn't thing of or such a thing.
>
> Do post the data, and I'm sure things will be sorted out soon.[/color]

What should I actually post?

Thank you very much Diez, I appreciate your help.

**Diez B. Roggisch** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

> What should I actually post?

The email text. Whatever the reason for the unexpected behaviour is, its in
there.
[color=blue]
> Thank you very much Diez, I appreciate your help.[/color]

Your welcome.

--
Regards,

Diez B. Roggisch

**Luka Milkovic** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

On Sun, 12 Sep 2004 16:24:31 +0200, Diez B. Roggisch wrote:
[color=blue]
> The email text. Whatever the reason for the unexpected behaviour is, its in
> there.[/color]

('+OK 4815 octets', ['Received: from galileo.resean
(aelthegrin@cmu ng149.cmu.carne t.hr [193.198.128.149])', '\tby
jagor.srce.hr (8.12.10/8.12.10) with ESMTP id i8BFvSRt009065' , '\tfor
<luka.milkovic@ public.srce.hr> ; Sat, 11 Sep 2004 17:57:28 +0200 (CEST)',
'Date: Sat, 11 Sep 2004 17:57:28 +0200 (CEST)', 'Message-Id:
<200409111557.i 8BFvSRt009065@j agor.srce.hr>', 'From:
luka.milkovic@p ublic.srce.hr', 'To: luka.milkovic@p ublic.srce.hr',
'Subject: OTP', 'X-Spam-Score: 5.544 (*****)
DATE_MISSING,MS GID_FROM_MTA_SH ORT,NO_REAL_NAM E', 'X-Scanned-By: MIMEDefang
2.42', 'X-Virus-Scanned: by amavisd-new at jagor.srce.hr',
'Content-Length: 4210', 'Status: ', '', '', '---Code block---', '[6964,
7086, 3211, 7522, 9472, 3265, 3610, 104, 9729, 6706, 8035, 5439, 7142,
360, 677, 1667, 1382, 9417, 4493, 8289, 9613, 3470, 889, 1021, 3381, 3480,
1385, 2027, 956, 9317, 6567, 5552, 1114, 3311, 4437, 631, 5881, 2101,
9948, 4529, 3088, 5548, 3728, 8727, 7787, 5754, 8315, 8250, 8308, 510,
8183, 4052, 9046, 8217, 5107, 8333, 7799, 4589, 209, 7465, 1010, 4459,
5984, 8272, 5311, 4458, 3565, 5747, 8460, 9845, 9305, 1662, 2650, 5290,
9725, 5743, 6679, 9896, 4776, 8586, 3075, 8824, 9369, 6957, 8564, 7165,
112, 9940, 6291, 1489, 3561, 1218, 3890, 9970, 9973, 7624, 7721, 8620,
456, 872, 4546, 926, 2687, 8884, 8598, 7544, 6857, 5363, 6686, 8579, 7937,
8290, 3578, 5411, 6375, 5596, 6860, 8392, 5300, 5927, 8211, 2232, 2194,
1388, 9047, 5384, 876, 4773, 7331, 3238, 5699, 7498, 2789, 8344, 5198,
1732, 3330, 6832, 908, 4210, 8943, 2390, 1655, 5324, 993, 6281, 2909,
2178, 9929, 40, 5060, 964, 4752, 8570, 7714, 607, 6450, 5793, 9292, 6428,
5410, 7567, 6040, 543, 3602, 8022, 4052, 7222, 6324, 6729, 1030, 299,
8641, 4312, 8614, 423, 6730', ', 6793, 3453, 9470, 9382, 2037, 4103,
6427, 5312, 1366, 6287, 2316, 5745, 6916, 1640, 2381, 7510, 1156, 1538,
3015, 1592, 4136, 2170, 6263, 3829, 6869, 8079, 9724, 1830, 3245, 4694,
782, 9703, 3615, 2907, 4435, 7329, 7511, 5418, 2913, 1567, 7865, 3729,
8289, 373, 5635, 8292, 9569, 4370, 8728, 3082, 7829, 4797, 9632, 8283,
2741, 7887, 6366, 9821, 1604, 1099, 3256, 2722, 8474, 6261, 8582, 6431,
1762, 8615, 9745, 599, 4078, 4779, 1469, 90, 5432, 5475, 9098, 5614, 184,
9515, 8909, 3868, 4880, 2408, 9665, 8552, 5444, 9209, 993, 9008, 1495,
1885, 3871, 4774, 8698, 5212, 1303, 6629, 6011, 4490, 9329, 1062, 4558,
4338, 2279, 8502, 473, 9650, 5787, 8329, 6816, 6858, 3868, 1854, 2991,
9958, 8931, 9276, 7837, 9372, 6732, 2402, 5453, 6012, 2958, 2593, 2258,
6599, 2127, 2214, 5839, 3947, 5270, 10093, 8043, 2905, 686, 6451, 312,
1682, 1947, 3447, 4083, 6838, 7896, 3054, 9913, 6716, 3831, 1861, 7286,
6863, 7754, 5534, 8451, 9536, 7945, 9747, 7075, 3808, 6180, 5387, 930,
9663, 7337, 3513, 9535, 4329, 6056, 2114, 8972, 8336, 9743, 5397, 3112,
8023, 3392, 1488, 1707, 8223, 9982, 4498, 1840, 962, 2471, 7919, 2731,
7935, 2826, 6904, 4150, 8780, 9697, 5955, 412, 1816, 7017, 5219, 1290,
7106, 6747, 1180, 1230, 2564, 1568, 373, 9301, 59, 9632, 4667, 7701, 9141,
6240, 3290, 7172, 4006, 8018, 5744, 1125, 4388, 7109, 7357, 5188, 841,
7950, 666, 6754, 4894, 7222, 9275, 7291, 3038, 6510, 8543, 7400, 2218,
2671, 1, 1753, 5620, 4833, 2920, 3754, 9364, 9724, 3445, 6378, 1986, 9350,
4887, 633, 6400, 4586, 1541, 5883, 2696, 306, 5971, 8164, 748, 2464, 550,
9843, 9373, 5004, 4295, 1055, 6916, 6386, 8480, 4480, 8744, 2586, ',
'6573, 869, 9277, 6960, 4871, 9340, 6119, 4271, 7572, 1230, 1213, 5534]',
'---Code block---'], 4815)

This is the original mail, sorry because of the size. As you can see,
there are two problematic spots: 6730', ', and ','6573, at the end of the
mail.

I was wandering is there any way to modify my splitting code I already
posted? The thing I want to implement is that the code would parse e-mail
as usual and when it comes to these problematic spots, it removes
unnecessary quotes and continues parsing...

Is there anything that could be done?

Thank you once more:)
Luka

**Diez B. Roggisch** · Jul 18 '05, 03:18 PM

Re: Mail extraction problem (something's wrong with split methods)

> I was wandering is there any way to modify my splitting code I already[color=blue]
> posted? The thing I want to implement is that the code would parse e-mail
> as usual and when it comes to these problematic spots, it removes
> unnecessary quotes and continues parsing...
>
> Is there anything that could be done?[/color]

Well, you could certainly code around these special cases - however, it
seems to me that whatever generates this mail is malfunctioning. Not on the
transport-layer, but from the thing that produces this

---Code block---

thingy.

What is that actually for? It looks as if you try to reinvent the wheel and
produce your own encoding scheme for binary data - instead of doing this, I
suggest you use one of the several available standards, like uuencode or
others. These are covered by standard apis in python as well as in other
languages. Better go for them.

--
Regards,

Diez B. Roggisch

**Pierre Fortin** · Jul 18 '05, 03:19 PM

Re: Mail extraction problem (something's wrong with split methods)

On Sun, 12 Sep 2004 17:32:15 +0200 Luka wrote:

This msg has already been processed by something that appears to generate
list/tuple segments... I would suspect that whatever modified the message
has a string size limitation... However, it looks like whatever
manhandled this msg just did what looks like a python print of a tuple...
If you really want to process this type of message instead of getting at
the real problem, then here's a clue...

Here, I reduced the contents to just the items...

('+OK',
['Received', # brackets, braces, parens are just text herein
'by',
'for',
'Date',
'Message-Id',
'From',
'To',
'Subject',
'X-Scanned-By: MIMEDefang 2.42',
'X-Virus-Scanned',
'Content-Length: 4210',
'Status: ',
'',
'',
'---Code block---',
'[6964, 7086, ..., 6730', # "[" is just text here
', 6793, ..., 5534]', # "]" ditto
'---Code block---'
],
4815
)

Further reducing the items shows the structure:

('s',
['s', 's', 's', 's', 's', 's', 's', 's', 's', 's', 's', 's', # headers
'', # header/body separator
'',
's', # ---Code block---
's', # 1st part
's', # 2nd part
's' # ---Code block---
],
4815 # reported msg size
)

which boils down to:

(s,[s, ..., s],i) # aka: tuple(string,li st(strings),int )

So... looks like you just need to isolate the strings between the
"---Code block---" strings (could be more than 2 or just 1) and
concatenate them. splitting the result...

Straight-line brute forcing it:

msg = .... # get the message as a tuple
sep = "---Code block---"
start = msg[1].index(sep)
data = msg[1][start+1:]
end = data.index(sep)
data = data[:end]
print "".join(dat a)[1:-1].split(", ")
[color=blue]
> This is the original mail, sorry because of the size. As you can see,
> there are two problematic spots: 6730', ', and ','6573, at the end of
> the mail.[/color]

Pierre

**Luka Milkovic** · Jul 18 '05, 03:19 PM

Re: Mail extraction problem (something's wrong with split methods)

On Sun, 12 Sep 2004 18:11:38 +0200, Diez B. Roggisch wrote:
[color=blue]
> Well, you could certainly code around these special cases - however, it
> seems to me that whatever generates this mail is malfunctioning. Not on the
> transport-layer, but from the thing that produces this
>
> ---Code block---
>
> thingy.[/color]

The thing that produces this Code block thingy is a /dev/random output.
I'm actually building a primitive One-time pad:) Here is the code (not
translated, but you'll understand I think):

duljina_grupe = 4
dev_random = open("/dev/random")
P = select.poll()
P.register(dev_ random.fileno() ,select.POLLIN)
grupe = []

while len(grupe) < gener:
grupa = ""
while len(grupa) < duljina_grupe:
if P.poll(0.1):
datum = ord(dev_random. read(1))
if datum < 200:
grupa+="%2.2d"% (datum%100)
else:
print "Nedovoljna kolicina entropije u /dev/random! Pomaknite misa"
print "Za sada generirano %d grupa"%len(grup e)
grupe.append(in t(grupa))

And overlapping it with text transfered in ASCII format:

grupe = []
for char in otv:
grupe.append(or d(char))
f2 = open("OTpad.pad ","r")
prvi = pickle.load(f2)
enc_txt = map(lambda x,y: (x or 0) ^ (y or 0),prvi,grupe)

This enc_txt is the thingy which is mailed... Do you see any mistakes?
[color=blue]
> What is that actually for? It looks as if you try to reinvent the wheel
> and produce your own encoding scheme for binary data - instead of doing
> this, I suggest you use one of the several available standards, like
> uuencode or others. These are covered by standard apis in python as well
> as in other languages. Better go for them.[/color]

[color=blue]
>Straight-line brute forcing it:[/color]
[color=blue]
>msg = .... # get the message as a tuple
>sep = "---Code block---"
>start = msg[1].index(sep)
>data = msg[1][start+1:]
>end = data.index(sep)
>data = data[:end]
>print "".join(dat a)[1:-1].split(", ")[/color]

Thanks Pierre, I'll try something like that later.

Luka

**Diez B. Roggisch** · Jul 18 '05, 03:20 PM

Re: Mail extraction problem (something's wrong with split methods)

> This enc_txt is the thingy which is mailed... Do you see any mistakes?

Nope - but the result of that operation should be a stream of ints stored in
a list. Thats ok - but how do you actually _create_ the mail text - do you
use the repr() of the list? That certainly is not a good idea, as that
representation is made for human readability.

Better to use e.g. module struct to create a string out of it, and uuencode
that.

The more interesting code is what you do with your list to actually create
the mail.

--
Regards,

Diez B. Roggisch

**Jeff Epler** · Jul 18 '05, 03:20 PM

Re: Mail extraction problem (something's wrong with split methods)

You cannot send data with arbitrary-length lines over SMTP without using
an encoding such as quoted-printable or base64.

RFC2821
section 2.3.7 "Limits MAY be imposed on line lengths by servers"
section 4.5.3:
The maximum total length of a text line including the <CRLF>
is 1000 characters (not counting the leading dot duplicated
for transparency). This number may be increased by the use
of SMTP Service Extensions.
(The relevant SMTP Service Extension in this case being RFC1652, 8BITMIME)

While this is not the limit you seem to be running into (you said
problems happened at 350 bytes) you should be aware of this (and lots of
other details about SMTP and other mail protocols) before you write
something to run "on top of" SMTP.

If you want to send a Python data structure across the network, you
might use pickle and then the email module to create a properly
MIME-encoded message:
import pickle, email.Message
bytes = pickle.dumps(my _structure)
message = email.Message.M essage()
message.set_typ e("applicatio n/x-luka-milkovic")
message.set_pay load(bytes.enco de("base64"))
# set other headers as needed
message = str(message)

If your structure is actually a sequence of fixed-width integers, then
you might be happy using struct.pack() instead of pickle:
import struct
l = len(my_structur e)
bytes = struct.pack("!" + "H"*l, *my_structure)
"H" is for numbers in range(65536), "!" uses network byte-order

In either case, you perform the reverse steps on the reassembled message
on the other end:
import email.Parser
decoded_message = email.Parser.Pa rser().parsestr (message)
decoded_message .get_type() # must be application/x-luka-milkovic
# or this is not a message from your
# program
bytes = decoded_message .get_payload(). decode("base64" )
# now pickle.loads or struct.unpack the bytes

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFBRZNuJd0 1MZaTXX0RApJ2AJ 9z7eBihqJNZfudV EE3LLx2ujPgBgCg kogk
SXZ4DMoU6CILzly m0gyt8mo=
=rdSM
-----END PGP SIGNATURE-----

**Luka Milkovic** · Jul 18 '05, 03:20 PM

Re: Mail extraction problem (something's wrong with split methods)

On Mon, 13 Sep 2004 07:32:47 -0500, Jeff Epler wrote:
[color=blue]
> In either case, you perform the reverse steps on the reassembled message
> on the other end:
> import email.Parser
> decoded_message = email.Parser.Pa rser().parsestr (message)
> decoded_message .get_type() # must be application/x-luka-milkovic
> # or this is not a message from your
> # program
> bytes = decoded_message .get_payload(). decode("base64" )
> # now pickle.loads or struct.unpack the bytes
>
> Jeff[/color]

Hi Jeff, thanks for the info about SMTP protocol, I knew I was doing
something wrong, and now I know what exactly went wrong. I was thinking
about encoding, but at the time of the development of the sending part, I
decided to postpone it, and that seems to be a big mistake.

But, you mentioned MIME types and the methods of creating MIME mails. I
was always afraid of MIME because I don't understand it well (though I've
read documentation). I did what you told me to do about sending e-mail,
and it works fine. But when it comes to decoding the mail, I have some
problems. I connect to my POP3 server and download my MIME e-mail (using
command pop.retr()) and save it to a file tempMail.dat. I don't know what
to do next, I tried with the parser but it doesn't work. My code looks
like the one you've given above.

print bytes command returns nothing, a blank line... I don't know what to
do, please help.

Thank you very much.

**Luka Milkovic** · Jul 18 '05, 03:22 PM

Re: Mail extraction problem (something's wrong with split methods)

Thanks everybody, I finally got things working. I bypassed MIME because,
as I already said, I don't understand it well, and used base64 module and
encoding... Everything is working as it should, thank you very much:)

Mail extraction problem (something's wrong with split methods)

Mail extraction problem (something's wrong with split methods)

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment