How can I extract text from a running Python script?

[Alligatorr]

First, I'm completely new to using python, but willing to learn. I have very little programming skills, but again, I'm willing to learn.

Second, I have a script (given to me) that receives data from a remote sensor. The data posts to a running python script, each data entry occupying one line.

I need to get the data out to log it, graph it, and ultimately overlay it onto an image. Is there a way to do this in real time?

I'm aware of the ability to save the data as a text file, then import it into Microsoft Excel, but this will not work in real time. My end goal is to see the data on a web site at any time.

Any suggestions?

[bartonc]

First, I'm completely new to using python, but willing to learn. I have very little programming skills, but again, I'm willing to learn.

Second, I have a script (given to me) that receives data from a remote sensor. The data posts to a running python script, each data entry occupying one line.

I need to get the data out to log it, graph it, and ultimately overlay it onto an image. Is there a way to do this in real time?

I'm aware of the ability to save the data as a text file, then import it into Microsoft Excel, but this will not work in real time. My end goal is to see the data on a web site at any time.

Any suggestions?

Hi there, Alligatorr.
You've come the right place, as we are more than willing to teach (we live for it).
While it is possible to get the "text" of a running script, it is not likely that a running script is being modified. Here's why:

• Python is compiled (invisibly) before it is executed.
• It is would be very poor design to modify a script if point 1 were not true.

If you can pick out the part of the script that looks like it is writing the data and paste it here, we can help you get at it in real time.

There are instructions for using [code] tags on the right hand side of the page when you reply.

Welcome,
Barton

[Alligatorr]

If you can pick out the part of the script that looks like it is writing the data and paste it here, we can help you get at it in real time.

Thanks Bartonc.

From what I can tell, it is simply a "print" command that displays the data on the screen. There is no fancy code involved.

Instead of writing it to the screen, I need to get it to something where I can graph it over time. That's the goal.

[bartonc]

Thanks Bartonc.

From what I can tell, it is simply a "print" command that displays the data on the screen. There is no fancy code involved.

Instead of writing it to the screen, I need to get it to something where I can graph it over time. That's the goal.

Printing actually calls write() on a file-like object called stdout.
Anything that can be printed (an excellent way of testing) can be logged simply by redirecting the output of the print statement with the >> operator:[CODE=python]from time import time, sleep

# simulated process data
rawdata = \
['01 0.00 3.67 0.00 0.00',
'02 0.00 0.00 3.67 0.00',
'03 0.00 0.00 0.00 3.67',
'04 0.00 3.67 0.00 0.00',
'05 3.67 0.00 0.00 0.00',
'06 3.46 0.00 0.22 0.00',
'07 0.00 0.00 3.67 0.00',
'08 0.00 0.00 0.00 3.67',
'09 0.00 0.00 0.00 3.67',
'10 0.00 3.67 0.00 0.00',
'11 3.67 0.00 0.00 0.00',
'12 3.67 0.00 0.00 0.00',
'13 0.00 0.00 3.67 0.00',
'14 0.00 0.00 0.00 3.67',
'15 0.00 0.00 3.67 0.00']

# create an empty file when the script starts
logfile = open('logfile.t xt', 'w')
logfile.close()
for data in rawdata:
# open the file in 'append' mode
logfile = open('logfile.t xt', 'a')
print >> logfile, time(),
print >> logfile, data
# flush the buffer
logfile.close()
sleep(1)[/CODE]logfile.txt:
1191531584.84 01 0.00 3.67 0.00 0.00
1191531585.84 02 0.00 0.00 3.67 0.00
1191531586.84 03 0.00 0.00 0.00 3.67
1191531587.84 04 0.00 3.67 0.00 0.00
1191531588.84 05 3.67 0.00 0.00 0.00
1191531589.84 06 3.46 0.00 0.22 0.00
1191531590.84 07 0.00 0.00 3.67 0.00
1191531591.84 08 0.00 0.00 0.00 3.67
1191531592.84 09 0.00 0.00 0.00 3.67
1191531593.84 10 0.00 3.67 0.00 0.00
1191531594.84 11 3.67 0.00 0.00 0.00
1191531595.84 12 3.67 0.00 0.00 0.00
1191531596.84 13 0.00 0.00 3.67 0.00
1191531597.84 14 0.00 0.00 0.00 3.67
1191531598.84 15 0.00 0.00 3.67 0.00

[Alligatorr]

I successfully modified my python script to save the text to a log file which I FTP to my web site. Now I need to figure out a way to graph the data. As you can see, the data is a series of two temperature readings. I need to convert the temperature to fahrenheit and somehow graph the data. Any ideas?

Also, I need to cut the log file off at say 1000 entries so it doesn't get too large. How do I do that?

[bvdet]

I successfully modified my python script to save the text to a log file which I FTP to my web site. Now I need to figure out a way to graph the data. As you can see, the data is a series of two temperature readings. I need to convert the temperature to fahrenheit and somehow graph the data. Any ideas?

Also, I need to cut the log file off at say 1000 entries so it doesn't get too large. How do I do that?

I can help some. To truncate your data:[code=Python]fn_input = 'log.txt'
# create a shorter data file
fn_output = 'log_output.txt '

lines = 1000

f1 = open(fn_input)
# you can process the data without writing to a file if you want
lineList = [f1.next() for _ in range(lines)]
f2 = open(fn_output, 'w')
f2.write(''.joi n(lineList))
f1.close()
f2.close()[/code]To create a list of temperature readings:[code=Python]readList = [float(s.split(' ,')[1].strip("' \n")) for s in lineList][/code]I will leave it to you to convert the temperature to Fahrenheit. This data was truncated at 20 lines:[code=Python]>>> for temp in readList:
... print temp
...
22.05
24.67
22.69
22.35
24.64
22.05
22.48
24.55
22.16
24.52
21.85
24.49
21.59
24.42
21.71
24.38
21.58
24.37
21.34
24.32[/code]

[Alligatorr]

Because I'm a beginner in Python, this is somewhat hard for me - but I'm enjoying learning.

Here is what I have thus far (this is only a portion of my code):

while 1:
    data, address = s.recvfrom(1024)
    if not data:
        break
    s.sendto(data, address)

    temperature = ((9 / 5) * repr(data[:-1])) + 32

    print time.strftime("%b %d %H:%M:%S ", time.localtime()), address[0], ":", temperature)
    logfile = open('logfile.txt', 'a')
    print >> logfile, time.strftime("%b %d %H:%M:%S ", time.localtime()), ":", temperature)
    logfile.close()

    fn_input = 'logfile.txt'
    fn_output = 'logfile_short.txt'
    lines = 1000
    f1 = open(fn_input)
    lineList = [f1.next() for _ in range(lines)]
    f2 = open(fn_output, 'w')
    f2.write(''.join(lineList))
    f1.close()
    f2.close()
s.close()

I didn't parse the temperature data because I'm still having trouble figuring out how to separate the temperature data from each sensor (#1 and #2 on the logfile.txt output).

1. Can someone help me make sure my mathmatical function will operate correctly to convert the temperature to Fahrenheit?

2. How can I separate the temperature data from each sensor into a separate log file?

3. Will my code for limiting the file size to 1000 lines work? I'm not at my home computer now to test this.

[bvdet]

Because I'm a beginner in Python, this is somewhat hard for me - but I'm enjoying learning.

Here is what I have thus far (this is only a portion of my code):

while 1:
    data, address = s.recvfrom(1024)
    if not data:
        break
    s.sendto(data, address)

    temperature = ((9 / 5) * repr(data[:-1])) + 32

    print time.strftime("%b %d %H:%M:%S ", time.localtime()), address[0], ":", temperature)
    logfile = open('logfile.txt', 'a')
    print >> logfile, time.strftime("%b %d %H:%M:%S ", time.localtime()), ":", temperature)
    logfile.close()

    fn_input = 'logfile.txt'
    fn_output = 'logfile_short.txt'
    lines = 1000
    f1 = open(fn_input)
    lineList = [f1.next() for _ in range(lines)]
    f2 = open(fn_output, 'w')
    f2.write(''.join(lineList))
    f1.close()
    f2.close()
s.close()

I didn't parse the temperature data because I'm still having trouble figuring out how to separate the temperature data from each sensor (#1 and #2 on the logfile.txt output).

1. Can someone help me make sure my mathmatical function will operate correctly to convert the temperature to Fahrenheit?

2. How can I separate the temperature data from each sensor into a separate log file?

3. Will my code for limiting the file size to 1000 lines work? I'm not at my home computer now to test this.

#2 - Following is one way to separate the data by sensor number. It creates a dictionary, and the sensor number is the dictionary key. The code handles the temperature conversion. From there, the data can be written to disc by key.[code=Python]import re
# limit data to 40 lines
lines = 40

def c_to_f(num):
return (num*5.0/9)+32.0

f1 = open(fn_input)
lineList = [f1.next() for _ in range(lines)]
f1.close()

patt = re.compile(r"'( .+)'")
readDict = {}
for item in lineList:
key, value = [s.strip() for s in patt.search(ite m).group(1).spl it(",")]
readDict.setdef ault(key, []).append(c_to_f (float(value)))[/code]You can also split lineList this way:[code=Python]sensor1List = [item for item in lineList if "'1," in item]
sensor2List = [item for item in lineList if "'2," in item][/code]#3 The code I provided will limit your data to the number of lines assigned to variable lines.

[Alligatorr]

Thanks bvdet, I really appreciate your help. I'm learning quite a bit, but I think you may have gone beyond my level of knowledge.

I'm not sure how to use a key with a dictionary. I've read up on dictionaries, but they seem a lot harder to use than the simple print commands that I've been using. Please keep in mind this is my first code ever in python.

I understand this part of my program:
[CODE=python]
# This converts the temperature to Fahrenheit
def c_to_f(num):
return ((num*9.0/5.0) + 32.0)
[/CODE]

I also understand the basic print commands to save text to a file. I just can't figure out how to add what you posted, or where to add it to what I have.

The really confusing parts involve lineList, the functions, and the dictionary.

Still, even though this is difficult, I'm enjoying it. Thanks for taking the time to help.

[bvdet]

Thanks bvdet, I really appreciate your help. I'm learning quite a bit, but I think you may have gone beyond my level of knowledge.

I'm not sure how to use a key with a dictionary. I've read up on dictionaries, but they seem a lot harder to use than the simple print commands that I've been using. Please keep in mind this is my first code ever in python.

I understand this part of my program:
[CODE=python]
# This converts the temperature to Fahrenheit
def c_to_f(num):
return ((num*9.0/5.0) + 32.0)
[/CODE]

I also understand the basic print commands to save text to a file. I just can't figure out how to add what you posted, or where to add it to what I have.

The really confusing parts involve lineList, the functions, and the dictionary.

Still, even though this is difficult, I'm enjoying it. Thanks for taking the time to help.

I am not very good at explanations. A good way to learn is to play with the code in an interactive environment. I am using Pythonwin. A dictionary is a mapping type object (hash table, association list, etc.).[code=Python]>>> readDict['1']
[45.705555555555 556, 45.688888888888 89, 45.638888888888 886, 45.622222222222 22, 45.605555555555 554, 45.566666666666 67, 45.544444444444 444, 45.538888888888 891, 45.511111111111 113, 45.477777777777 781, 45.455555555555 556, 45.416666666666 664, 45.388888888888 886, 45.327777777777 776, 45.299999999999 997, 45.266666666666 666, 45.238888888888 887, 45.200000000000 003, 45.177777777777 777]
>>> readDict['2']
[44.25, 44.605555555555 554, 44.416666666666 664, 44.25, 44.488888888888 887, 44.311111111111 11, 44.138888888888 886, 43.994444444444 447, 44.061111111111 11, 43.988888888888 887, 43.855555555555 554, 43.75, 43.649999999999 999, 43.566666666666 663, 43.427777777777 777, 43.361111111111 114, 43.305555555555 557, 43.233333333333 334, 43.172222222222 224, 43.111111111111 114, 43.061111111111 11]
>>> [/code][code=Python]for key in readDict:
print "Sensor #%s:" % key
for item in readDict[key]:
print " %.4f" % item[/code]
Partial output:

Sensor #1:
45.7056
45.6889
45.6389
45.6222
45.6056
...........
Sensor #2:
44.2500
44.6056
44.4167
44.2500
44.4889
...........

Following is another way of getting the number of desired lines from the data file and creating a list of those lines (lineList):[code=Python]>>> lineList = []
>>> f1 = open(fn_input)
>>> lines = 10
>>> for i in range(lines):
... lineList.append (f1.readline())
...
>>> for line in lineList:
... print line,
...
Nov 28 17:34:23 76.17.211.244 : '2, 22.05'
Nov 28 17:35:38 76.17.211.244 : '1, 24.67'
Nov 28 17:39:23 76.17.211.244 : '2, 22.69'
Nov 28 17:44:23 76.17.211.244 : '2, 22.35'
Nov 28 17:45:38 76.17.211.244 : '1, 24.64'
Nov 28 17:49:23 76.17.211.244 : '2, 22.05'
Nov 28 17:54:23 76.17.211.244 : '2, 22.48'
Nov 28 17:55:38 76.17.211.244 : '1, 24.55'
Nov 28 17:59:23 76.17.211.244 : '2, 22.16'
Nov 28 18:00:38 76.17.211.244 : '1, 24.52'
>>> f1.close()[/code]In this case I used readline() instead of next() and compiled the list without using a list comprehension. HTH

[Alligatorr]

I'm trying to understand dictionaries, but I'm having a hard time.

I understand bits and pieces of what you have posted, but I don't know how to put all of the code together. Most of what I'm now doing is just guessing.

This is what I have thus far - any of it look right?

[CODE=python] # This converts the temperature to Fahrenheit
def c_to_f(num):
return ((num*9.0/5.0) + 32.0)

#This separates the data by sensor number, creates a dictionary, and the sensor number is the key
import re
# limit data to 100 lines
lines = 100

f1 = open(fn_input)
lineList = [f1.next() for _ in range(lines)]
f1.close()

patt = re.compile(r"'( .+)'")
readDict = {}
for item in lineList:
key, value = [s.strip() for s in patt.search(ite m).group(1).spl it(",")]
readDict.setdef ault(key, []).append(c_to_f (repr(data[:-1]))

# This writes key 1 to a logfile
logfile1 = open('sensor1_v alue.txt', 'a')
print >> logfile1, readDict["1"]
logfile1.close( )

# This writes key 2 to a logfile
logfile2 = open('sensor2_v alue.txt', 'a')
print >> logfile2, readDict["2"]
logfile2.close( )[/CODE]

[bvdet]

I'm trying to understand dictionaries, but I'm having a hard time.

I understand bits and pieces of what you have posted, but I don't know how to put all of the code together. Most of what I'm now doing is just guessing.

This is what I have thus far - any of it look right?

[CODE=python] # This converts the temperature to Fahrenheit
def c_to_f(num):
return ((num*9.0/5.0) + 32.0)

#This separates the data by sensor number, creates a dictionary, and the sensor number is the key
import re
# limit data to 100 lines
lines = 100

f1 = open(fn_input)
lineList = [f1.next() for _ in range(lines)]
f1.close()

patt = re.compile(r"'( .+)'")
readDict = {}
for item in lineList:
key, value = [s.strip() for s in patt.search(ite m).group(1).spl it(",")]
readDict.setdef ault(key, []).append(c_to_f (repr(data[:-1]))

# This writes key 1 to a logfile
logfile1 = open('sensor1_v alue.txt', 'a')
print >> logfile1, readDict["1"]
logfile1.close( )

# This writes key 2 to a logfile
logfile2 = open('sensor2_v alue.txt', 'a')
print >> logfile2, readDict["2"]
logfile2.close( )[/CODE]

The value you are passing to c_to_f is incorrect, and you have some indentation errors. Instead of passing a list to print and redirecting to a file, you can string format each element and write the formatted elements to the file. I made some modifications as follows.[code=Python]# This converts the temperature to Fahrenheit
def c_to_f(num):
return ((num*9.0/5.0) + 32.0)

#This separates the data by sensor number, creates a dictionary, and the sensor number is the key
import re
# limit data to 100 lines
lines = 100

f1 = open(fn_input)
lineList = [f1.next() for _ in range(lines)]
f1.close()

patt = re.compile(r"'( .+)'")
readDict = {}
for item in lineList:
key, value = [s.strip() for s in patt.search(ite m).group(1).spl it(",")]
readDict.setdef ault(key, []).append(c_to_f (float(value)))

# This writes key 1 to a logfile
logfile1 = open(r'H:\TEMP\ temsys\sensor1_ value.txt', 'a')
logfile1.write( "\n".join(['%.4f' % item for item in readDict["1"]]))
logfile1.close( )

# This writes key 2 to a logfile
logfile2 = open(r'H:\TEMP\ temsys\sensor2_ value.txt', 'a')
logfile2.write( "\n".join(['%.4f' % item for item in readDict["2"]]))
logfile2.close( )[/code]

[Alligatorr]

For some reason, the code is not working. It prints the data on the screen just fine, but doesn't write anything to a log file. When I just print "data" when the code is running, I get a result like '2, 23.70'. Does this mean I need to break this apart first?

Can anyone help? I'm very close!

Here is the main part of my code (based on what was posted earlier):
[CODE=python]
# This converts the temperature to Fahrenheit
def c_to_f(num):
return ((num*9.0/5.0) + 32.0)

while 1:
data, address = s.recvfrom(1024 )
if not data:
break
s.sendto(data, address)
print time.strftime(" %b %d %H:%M:%S ", time.localtime( )), address[0], ":", repr(data[:-1])
s.close()

# limit data to 100 lines
fn_input = 'logfile.txt'
fn_output = 'logfile_short. txt'
lines = 100

f1 = open(fn_input)
lineList = [f1.next() for _ in range(lines)]
f1.close()

patt = re.compile(r"'( .+)'")
readDict = {}
for item in lineList:
key, value = [s.strip() for s in patt.search(ite m).group(1).spl it(",")]
readDict.setdef ault(key, []).append(c_to_f (float(repr(dat a[:-1]))))

# This writes key 1 to a logfile
logfile1 = open(r'C:\Docum ents and Settings\User\M y Documents\Pytho n Scrips\sensor1_ value.txt', 'a')
logfile1.write( "\n".join(['%.4f' % item for item in readDict["1"]]))
logfile1.close( )

# This writes key 2 to a logfile
logfile2 = open(r'C:\Docum ents and Settings\User\M y Documents\Pytho n Scrips\sensor2_ value.txt', 'a')
logfile2.write( "\n".join(['%.4f' % item for item in readDict["2"]]))
logfile2.close( )
[/CODE]

[rogerlew]

Here a "batteries included" approach using numpy and pylab

[code=python]
import numpy
import pylab
# see http://matplotlib.sour ceforge.net/

from time import mktime
from datetime import datetime
# see http://pleac.sourcefor ge.net/pleac_python/datesandtimes.h tml

def c2f(strnum) : return (float(strnum)* 5.0/9)+32.0

# dict to convert month abbreviations to int representations
months={'Jan':1 ,'Feb':2,'Mar': 3,'Apr':4,'May' :5,'Jun':6,
'Jul':7,'Aug':8 ,'Sep':9,'Oct': 10,'Nov':11,'De c':12}

# list of dicts of lists to hold data
sensors=[{'time':[],'ip':[],'temp':[]},{'time':[],'ip':[],'temp':[]}]

# open file
fid=open('logfi le.txt','r')

# the [-100:] indexing gives the last hundred entries
for i, splitline in enumerate([line.split() for line in fid.readlines()[-100:]]):
tnow=datetime(2 007, # year not in log file
months[splitline[0]], # month as int
int(splitline[1]), # date
int(splitline[2][0:2]), # hour
int(splitline[2][3:5]), # minute
int(splitline[2][6:8]) ) # second
if i == 0: t0=tnow # save first time

sensors[int(splitline[5][1])-1]['time'].append(mktime( tnow.timetuple( ))-
mktime(t0.timet uple()))
sensors[int(splitline[5][1])-1]['ip'].append(splitli ne[3] )
sensors[int(splitline[5][1])-1]['temp'].append(c2f(spl itline[6][:-1]))

fid.close()

# find temp ensemble average
# we need to interpolate temp arrays first
s1t,s2t=numpy.a rray(sensors[0]['time']),numpy.array(s ensors[1]['time'])
ti=numpy.arange (0.,numpy.max(n umpy.concatenat e((s1t,s2t))),1 .)
s1ti=pylab.stin eman_interp(ti, sensors[0]['time'],sensors[0]['temp'])
s2ti=pylab.stin eman_interp(ti, sensors[1]['time'],sensors[1]['temp'])
temp_ave=(s1ti+ s2ti)/2.

# make plot
pylab.figure()
s1,=pylab.plot( sensors[0]['time'],sensors[0]['temp'])
s2,=pylab.plot( sensors[1]['time'],sensors[1]['temp'])
av,=pylab.plot( ti,temp_ave)
pylab.legend((s 1,s2,av),('Sens or 1','Sensor 2','Average'))
pylab.title('Te mperature from %s to %s'%(t0,tnow))
pylab.ylabel('T emperature in Fahrenheit')
pylab.xlabel('T ime in seconds since %s'%t0)
pylab.savefig(' TempTime.png')
pylab.show()
[/code]
Plot can be viewed here:

http://www.flickr.com/photos/rogerlew/2109181749/