Parsing data from pyserial

[Lone Wolf]

I'm trying to get data through my serial port from a CMUcam.
This gizmo tracks a color and returns a packet of data. The
packet has nine data points (well, really eight since the first
point is just a packet header) separated by spaces as follows: M
xxx xxx xxx xxx xxx xxx xxx xxx

Here is the code I am using (python v24):

import serial

ser=serial.Seri al('com1',baudr ate=115200, bytesize=8,
parity='N', stopbits=1,xonx off=0, timeout=1)

ser.write("PM 1") #This sets the CMUcam to poll mode

for i in range(0,100,1):
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)
print reading
components = reading.split()
print components
ser.close

Here is an example output:

M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']

My problem is that I am trying to get each data point of the
packet into a separate variable. Ordinarily, this would be easy,
as I would just parse the packet, read the array and assign each
element to a variable eg. mx = components[1]. However, that
doesn't work here because the original packet and the array that
I got from using the split() method are different. If I were to
try read the array created in the first example output, mx would
be 123 instead of 37 like it is in the packet. In the second
example, the array is 85 while the packet is 38.

As near as I can figure out, pyserial is reading a stream of
data and helpfully rearranging it so that it fits the original
packet format M xxx xxx xxx xxx xxx xxx xxx xxx. I would have
thought the split() method that I used on original packet (ie
the "reading" variable) would have just returned an array with
nine elements like the packet has. This is not the case, and I
am at a loss about how to fix this.

I've searched the archive here and elsewhere with no luck. Any
help REALLY appreciated!

Wolf :)

_______________ _______________ _______________ ___
Get your own "800" number
Voicemail, fax, email, and a lot more

http://www.ureach.com/reg/tag

[John Machin]

Re: Parsing data from pyserial

Lone Wolf wrote:

I'm trying to get data through my serial port from a CMUcam.
This gizmo tracks a color and returns a packet of data. The
packet has nine data points (well, really eight since the first
point is just a packet header) separated by spaces as follows: M
xxx xxx xxx xxx xxx xxx xxx xxx
>
Here is the code I am using (python v24):
>
import serial
>
ser=serial.Seri al('com1',baudr ate=115200, bytesize=8,
parity='N', stopbits=1,xonx off=0, timeout=1)
>
ser.write("PM 1") #This sets the CMUcam to poll mode
>
for i in range(0,100,1):
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)

You are asking for 40 bytes of data. You will get 40 bytes of data.

However your packets are (presumably) variable length, (presumably)
terminated by CR and/or LF. What does the documentation for the device
tell you?

print reading

What you see from the print statement is not necessarily what you've
got.
Change that to print repr(reading) and show us what you then see.

components = reading.split()
print components
ser.close
>
Here is an example output:
>
M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']
>

Let's try to reconstruct "reading":

| >>a = ['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
| ... '124', '86', '25', 'M']
| >>astrg = ' '.join(a)
| >>astrg
| '59 123 87 25 M 37 79 3 4 59 124 86 25 M'
| >>len(astrg)
| 39 <<<<<==== ooh! almost 40!!
| >>b = ['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124',
'86',
| ... '25', 'M', '38'
| ... , '7']
| >>bstrg = ' '.join(b)
| >>bstrg
| '39 85 26 M 38 77 3 2 59 124 86 25 M 38 7'
| >>len(bstrg)
| 40 <<<<<==== ooh! exactly 40!!!

My guess: the device is pumping out packets faster than you can handle
them. So you are getting 40-byte snatches of bytes. A snatch is long
enough to cover a whole packet with possible fragments of packets at
each end. You will need to discard the fragments. If you need all the
data, you'd better get some help on how to implement flow control --
I've never used pyserial and I'm not going to read _all_ the docs for
you :-)

I'm very interested to see what print repr(reading) actually shows. I'm
strongly suspecting there is a CR (no LF) at the end of each packet; in
the two cases shown, this would cause the "print reading" to appear as
only one packet ... think about it: carriage return, with no linefeed,
would cause overwriting. It is a coincidence with those two samples
that the first part of the line doesn't appear strange, with a 4, 5, or
6-digit number showing up where the trailing fragment ends

My problem is that I am trying to get each data point of the
packet into a separate variable. Ordinarily, this would be easy,
as I would just parse the packet, read the array and assign each
element to a variable eg. mx = components[1].

better would be:

mx, foo, bar, ......, eighth_vbl = components[start:start + 8]
once you have worked out what start should be, e.g. start =
components.inde x('M') + 1

However, that
doesn't work here because the original packet and the array that
I got from using the split() method are different. If I were to
try read the array created in the first example output, mx would
be 123 instead of 37 like it is in the packet. In the second
example, the array is 85 while the packet is 38.
>
As near as I can figure out, pyserial is reading a stream of
data and helpfully rearranging it so that it fits the original
packet format M xxx xxx xxx xxx xxx xxx xxx xxx.

How, if you've read the docstring for the Serial.read() method, did you
come to that conclusion?

pyserial knows nothing about your packet format.

I would have
thought the split() method that I used on original packet (ie
the "reading" variable) would have just returned an array with
nine elements like the packet has. This is not the case, and I
am at a loss about how to fix this.
>
I've searched the archive here and elsewhere with no luck. Any
help REALLY appreciated!
>

With a bit of repr() and a bit of RTFM, one can often manage without
help :-)

Cheers,
John

[Giovanni Bajo]

Re: Parsing data from pyserial

Lone Wolf wrote:

reading = ser.read(40)

Simply try ser.readline() here, or maybe ser.readline(eo l="\r").

--
Giovanni Bajo

[Grant Edwards]

Re: Parsing data from pyserial

On 2006-12-03, Lone Wolf <lone_wolf@urea ch.comwrote:

import serial
>
ser=serial.Seri al('com1',baudr ate=115200, bytesize=8,
parity='N', stopbits=1,xonx off=0, timeout=1)
>
ser.write("PM 1") #This sets the CMUcam to poll mode
>
for i in range(0,100,1):
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)
print reading
components = reading.split()
print components
ser.close
>
Here is an example output:
>
M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']
>
My problem is that I am trying to get each data point of the
packet into a separate variable. Ordinarily, this would be
easy, as I would just parse the packet, read the array and
assign each element to a variable eg. mx = components[1].
However, that doesn't work here because the original packet
and the array that I got from using the split() method are
different.

I doubt it. Try printing `reading` instead of reading. I
suspect that the string you're getting from ser.read() has a
carraige-return in it that you aren't seeing when you do print
reading.

If I were to try read the array created in the first example
output, mx would be 123 instead of 37 like it is in the
packet. In the second example, the array is 85 while the
packet is 38.
>
As near as I can figure out, pyserial is reading a stream of
data and helpfully rearranging it so that it fits the original
packet format M xxx xxx xxx xxx xxx xxx xxx xxx.

No, it isn't. I wrote the Posix low-level code that's in
pyserial. I've used pyserial extensively on both Windows and
Linux. It doesn't rearrange anything.

I would have thought the split() method that I used on
original packet (ie the "reading" variable) would have just
returned an array with nine elements like the packet has. This
is not the case, and I am at a loss about how to fix this.

When something odd seems to be happening with strings, always
print `whatever` rather than whatever

I've searched the archive here and elsewhere with no luck. Any
help REALLY appreciated!

--
Grant Edwards grante Yow! There's a SALE on
at STRETCH SOCKS down at the
visi.com "7-11"!!

[Si Ballenger]

Re: Parsing data from pyserial

On Sat, 2 Dec 2006 23:02:06 -0500, Lone Wolf
<lone_wolf@urea ch.comwrote:

>I'm trying to get data through my serial port from a CMUcam.
>This gizmo tracks a color and returns a packet of data. The
>packet has nine data points (well, really eight since the first
>point is just a packet header) separated by spaces as follows: M
>xxx xxx xxx xxx xxx xxx xxx xxx
>
>Here is the code I am using (python v24):
>
>import serial
>
>ser=serial.Ser ial('com1',baud rate=115200, bytesize=8,
>parity='N', stopbits=1,xonx off=0, timeout=1)
>
>ser.write("P M 1") #This sets the CMUcam to poll mode
>
>for i in range(0,100,1):
> ser.write("TC 016 240 100 240 016 240\r\n")
> reading = ser.read(40)
> print reading
> components = reading.split()
> print components
>ser.close

In my dealing with serial gizmos I have to put a delay between
the request sent to the gizmo and the reading of the serial input
buffer for returned data. Serial ports and gizmos need some time
to do their thing.

[Grant Edwards]

Re: Parsing data from pyserial

On 2006-12-03, Si Ballenger <shb*NO*SPAM*@c omporium.netwro te:

In my dealing with serial gizmos I have to put a delay between
the request sent to the gizmo and the reading of the serial input
buffer for returned data. Serial ports and gizmos need some time
to do their thing.

I doubt that's the issue. He's reading with a 1-second timeout
value.

--
Grant Edwards grante Yow! The Korean War must
at have been fun.
visi.com

[Si Ballenger]

Re: Parsing data from pyserial

On Sun, 03 Dec 2006 16:52:33 -0000, Grant Edwards
<grante@visi.co mwrote:

>On 2006-12-03, Si Ballenger <shb*NO*SPAM*@c omporium.netwro te:
>

>In my dealing with serial gizmos I have to put a delay between
>the request sent to the gizmo and the reading of the serial input
>buffer for returned data. Serial ports and gizmos need some time
>to do their thing.

>
>I doubt that's the issue. He's reading with a 1-second timeout
>value.

I would think a time delay would be needed between the below two
lines in the code if he expects to get a useable data string back
from the gizmo for the command sent to it.

ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)

[John Machin]

Re: Parsing data from pyserial

Grant Edwards wrote:

When something odd seems to be happening with strings, always
print `whatever` rather than whatever
>

:-)

Unholy perlism, Batman!

For the benefit of gentle readers who are newish and might not have
seen the ` character in Python code outside a string literal, or for
those who'd forgotten, there is a cure:

| >>re.sub(r"`(.* ?)`", r"repr(\1)", "print `whatever`, `foo`, `bar`")
| 'print repr(whatever), repr(foo), repr(bar)'


:-)

[Grant Edwards]

Re: Parsing data from pyserial

On 2006-12-03, Si Ballenger <shb*NO*SPAM*@c omporium.netwro te:

>>In my dealing with serial gizmos I have to put a delay between
>>the request sent to the gizmo and the reading of the serial input
>>buffer for returned data. Serial ports and gizmos need some time
>>to do their thing.

>>
>>I doubt that's the issue. He's reading with a 1-second timeout
>>value.

>
I would think a time delay would be needed between the below two
lines in the code if he expects to get a useable data string back
from the gizmo for the command sent to it.
>
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)

No. A delay isn't needed as long as the device responds within
1 second. The read() call will wait up to 1 second for the
first byte of the response.

--
Grant Edwards grante Yow! NEWARK has been
at REZONED!! DES MOINES has
visi.com been REZONED!!

[Grant Edwards]

Re: Parsing data from pyserial

On 2006-12-03, John Machin <sjmachin@lexic on.netwrote:

Grant Edwards wrote:
>

>When something odd seems to be happening with strings, always
>print `whatever` rather than whatever
>>

>
>:-)
>
Unholy perlism, Batman!

OK, make that "print repr(whatever)" . :)

--
Grant Edwards grante Yow! I selected E5... but
at I didn't hear "Sam the Sham
visi.com and the Pharoahs"!

[Fredrik Lundh]

Re: Parsing data from pyserial

Si Ballenger wrote:

I would think a time delay would be needed between the below two
lines in the code if he expects to get a useable data string back
from the gizmo for the command sent to it.
>
ser.write("TC 016 240 100 240 016 240\r\n")
reading = ser.read(40)

why's that? if the gizmo is busy "doing its thing", read() will wait
for up to one second before giving up.

</F>

[Si Ballenger]

Re: Parsing data from pyserial

On Sun, 03 Dec 2006 18:44:07 -0000, Grant Edwards
<grante@visi.co mwrote:

>On 2006-12-03, Si Ballenger <shb*NO*SPAM*@c omporium.netwro te:
>

>>>In my dealing with serial gizmos I have to put a delay between
>>>the request sent to the gizmo and the reading of the serial input
>>>buffer for returned data. Serial ports and gizmos need some time
>>>to do their thing.
>>>
>>>I doubt that's the issue. He's reading with a 1-second timeout
>>>value.

>>
>I would think a time delay would be needed between the below two
>lines in the code if he expects to get a useable data string back
>from the gizmo for the command sent to it.
>>
> ser.write("TC 016 240 100 240 016 240\r\n")
> reading = ser.read(40)

>
>No. A delay isn't needed as long as the device responds within
>1 second. The read() call will wait up to 1 second for the
>first byte of the response.

Per what was posted (below), it appears that the the appropriate
data is being received. It may be possible that the cam may be
sending in a mode that is not in alignment with the binary
transmission mode of the serial port. As a test I'd jumper
between the Tx and Rx pin on the serial port and then send out
the "M" line being received, then see if it will parse as
expected.

Here is an example output:

M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']

[John Machin]

Re: Parsing data from pyserial


Si Ballenger wrote:

>
Per what was posted (below), it appears that the the appropriate
data is being received.

[snip]

>
Here is an example output:
>
M 37 79 3 4 59 124 86 25
['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
'124', '86', '25', 'M
']
M 38 77 3 2 59 124 86 25
['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
'25', 'M', '38'
, '7']

Based on the split() results (presumably much more reliable than the
"print reading" results) what appears to me is:
fragment '59', '123', '87', '25'
packet 'M', '37', '79', '3', '4', '59', '124', '86', '25'
fragment 'M', '39' [see note]
fragment '85', '26'
packet 'M', '38', '77', '3', '2', '59', '124', '86', '25',
fragment 'M', '38'

[note] the 39 obviously aligns with the 37 and 38s, not with the 123
and 124s. However the boundary of the 2 split() results lies before the
39, not after. Puzzling.

In any case, I wouldn't call that "the appropriate data is being
received" -- looks like chunks missing to me.

[Si Ballenger]

Re: Parsing data from pyserial

On 3 Dec 2006 17:33:59 -0800, "John Machin"
<sjmachin@lexic on.netwrote:

>In any case, I wouldn't call that "the appropriate data is being
>received" -- looks like chunks missing to me.

Well, below is the posted expected return data format from the
cam and below that is what has been reported to be returned from
the cam when it is polled, which appears to be a fairly
reasonable match. I assume that each xxx is a decimal number
repersenting a single byte. In the binary mode each x in the
string might be considered a byte in itelf and possibly evaluated
as such. Anyhow it should be easy to see if the received string
can be parsed on it own correctly when not being received via the
serial port. That would start to narrow down where something not
understood is comming into play.

M xxx xxx xxx xxx xxx xxx xxx xxx
M 37 79 3 4 59 124 86 25

[John Machin]

Re: Parsing data from pyserial


Si Ballenger wrote:

On 3 Dec 2006 17:33:59 -0800, "John Machin"
<sjmachin@lexic on.netwrote:
>

In any case, I wouldn't call that "the appropriate data is being
received" -- looks like chunks missing to me.

>
Well, below is the posted expected return data format from the
cam and below that is what has been reported to be returned from
the cam when it is polled, which appears to be a fairly
reasonable match. I assume that each xxx is a decimal number
repersenting a single byte. In the binary mode each x in the
string might be considered a byte in itelf and possibly evaluated
as such. Anyhow it should be easy to see if the received string
can be parsed on it own correctly when not being received via the
serial port. That would start to narrow down where something not
understood is comming into play.
>
M xxx xxx xxx xxx xxx xxx xxx xxx
M 37 79 3 4 59 124 86 25

Try reading previous posts. The OP reported that to be returned from
the cam, based on print forty_bytes, not print repr(forty_byte s). I
think everybody (including possibly even the OP) is willing to believe
that the cam is *generating* correct parseable stuff, followed by '\r'
-- the problem now is how to get as many samples per second as is
reasonable in the face of problems like lack of buffering, flow
control, etc.