File Parsing

**psbasha** · Aug 10 '07, 07:43 PM

Originally posted by psbasha

Hi ,

Below is the file format ,which has Keywords in the file.I would like to store the data in the different variables ( Parameters,Poin ts ,Lines ,Circle)

Code:

Sample.txt

$$$$Header$$$$$$$$$$$$
$$$$Parameter$$$$$$$$$

/Parameter_Value/ 1.0

/Point/ 
10.0 10.0 10.0 $ Comment: Point Data
20.0 20.0 20.0 

$$$$$$Line$$$$$$$$

/Line/ $Line Data

10.0 15.0 0.0
20.0 10.0 0.0 

$$$$$$Circle$$$$$$$$
/Circle/

10.0 $Radius

0.0 0.0 0.0  $Center

Can body help me in the best way ( Oprtimized way - interms of lines of code) of writing the code.

Thanks
PSB

The above is a sample data only.I have to read different Unique Geometry elements data in that file format having the different and unique key word

-PSB

**bvdet** · Aug 11 '07, 03:16 AM

Originally posted by psbasha

The above is a sample data only.I have to read different Unique Geometry elements data in that file format having the different and unique key word

-PSB

It's probably not the best way, but it seems to work. All dictionary values are lists:[code=Python]import re

key_patt = re.compile(r'/([A-Za-z_]+)/')
data_patt = re.compile(r'\d +.\d+')
fn = 'data.txt'

f = open(fn)
key = None
dd = {}
lineList = [line.strip() for line in open(fn).readli nes() \
if line != '\n' and not line.startswith ('$')]
for line in lineList:
try:
line = line[:line.index('$' )]
except:
pass
m = key_patt.search (line)
if m:
key = m.group(1)
dd[key] = []
if data_patt.searc h(line):
dd[key] = [float(data_patt .search(line).g roup(0))]
else:
dd[key] = []
else:
m1 = data_patt.searc h(line)
if m1:
dd[key].append([float(n) for n in data_patt.finda ll(line)])

for key in dd:
print '%s = %s' % (key, dd[key])[/code]Did you ever resolve the point translation issue (this thread )? You never responded after I posted what I thought was a solution for you. A little feedback would be appreciated. Here's the output:

>>> Line = [[10.0, 15.0, 0.0], [20.0, 10.0, 0.0]]
Parameter_Value = [1.0]
Circle = [[10.0], [0.0, 0.0, 0.0]]
Point = [[10.0, 10.0, 10.0], [20.0, 20.0, 20.0]]
>>>

**psbasha** · Aug 11 '07, 03:28 PM

Originally posted by bvdet

It's probably not the best way, but it seems to work. All dictionary values are lists:[code=Python]import re

key_patt = re.compile(r'/([A-Za-z_]+)/')
data_patt = re.compile(r'\d +.\d+')
fn = 'data.txt'

f = open(fn)
key = None
dd = {}
lineList = [line.strip() for line in open(fn).readli nes() \
if line != '\n' and not line.startswith ('$')]
for line in lineList:
try:
line = line[:line.index('$' )]
except:
pass
m = key_patt.search (line)
if m:
key = m.group(1)
dd[key] = []
if data_patt.searc h(line):
dd[key] = [float(data_patt .search(line).g roup(0))]
else:
dd[key] = []
else:
m1 = data_patt.searc h(line)
if m1:
dd[key].append([float(n) for n in data_patt.finda ll(line)])

for key in dd:
print '%s = %s' % (key, dd[key])[/code]Did you ever resolve the point translation issue (this thread )? You never responded after I posted what I thought was a solution for you. A little feedback would be appreciated. Here's the output:

>>> Line = [[10.0, 15.0, 0.0], [20.0, 10.0, 0.0]]
Parameter_Value = [1.0]
Circle = [[10.0], [0.0, 0.0, 0.0]]
Point = [[10.0, 10.0, 10.0], [20.0, 20.0, 20.0]]
>>>

Thanks BV for the solution.
The Point translation problem I have took the portion of the code snippet and solved with your approach.But if you have better approach than previous one,you can post the solution.So that I can use that approach.

-PSB

**psbasha** · Aug 24 '07, 06:02 PM

Hi,

I have the below file format,how to read in a concise way,
The file looke like this

Code:

Sample Data
4 Types
_up,1
_low,2
_left,5
_right,6

2Flags
_low,no
_up,yes

1 Data
x, 10

4 Values
1,0,0
1,1,0
1,1,1
1,1,0

2 Planes Type-1
1,0,0
0,0,0
0,1,0
0,0,0
0,1,0
0,0,1

In case of plane there are 2 planes defined,we have to have 2plane data seperate.

Thanks
PSB

**psbasha** · Sep 14 '07, 05:25 PM

Originally posted by bvdet

It's probably not the best way, but it seems to work. All dictionary values are lists:[code=Python]import re

key_patt = re.compile(r'/([A-Za-z_]+)/')
data_patt = re.compile(r'\d +.\d+')
fn = 'data.txt'

f = open(fn)
key = None
dd = {}
lineList = [line.strip() for line in open(fn).readli nes() \
if line != '\n' and not line.startswith ('$')]
for line in lineList:
try:
line = line[:line.index('$' )]
except:
pass
m = key_patt.search (line)
if m:
key = m.group(1)
dd[key] = []
if data_patt.searc h(line):
dd[key] = [float(data_patt .search(line).g roup(0))]
else:
dd[key] = []
else:
m1 = data_patt.searc h(line)
if m1:
dd[key].append([float(n) for n in data_patt.finda ll(line)])

for key in dd:
print '%s = %s' % (key, dd[key])[/code]Did you ever resolve the point translation issue (this thread )? You never responded after I posted what I thought was a solution for you. A little feedback would be appreciated. Here's the output:

>>> Line = [[10.0, 15.0, 0.0], [20.0, 10.0, 0.0]]
Parameter_Value = [1.0]
Circle = [[10.0], [0.0, 0.0, 0.0]]
Point = [[10.0, 10.0, 10.0], [20.0, 20.0, 20.0]]
>>>

Hi BV,

I have tried with above piece of code for reading some more filed formats as mentioned below ,the peice of code is not supporting this field format.Can you please suggest how to group for digits and alphanumeric values for the below scenarios.

Code:

Sample.txt
Sample.txt
 
$$$$Header$$$$$$$$$$$$
$$$$Parameter$$$$$$$$$
 
/Parameter_Value/ 1.0
 
/Point/ 
10.0 10.0 10.0 $ Comment: Point Data
20.0 20.0 20.0 
 
$$$$$$Line$$$$$$$$
 
/Line/ $Line Data
 
10.0 15.0 0.0
20.0 10.0 0.0 
 
$$$$$$Circle$$$$$$$$
/Circle/
 
10.0 $Radius
 
0.0 0.0 0.0  $Center


/DashedLineType/			21 $Dashed Line

/XMin_XMax_YMin_YMax/		1 27 1 37 $ Min and Max value

/LineFlag/		yes $ Flag to update


/XY-Plane/ 'Planes'
1,0,0
0,1,0
0,0,0

/XY-Plane/ 'Planes'
2,0,0
0,2,0
0,0,0


/Format/
$Values    
    3     3     1    50    25    28   'Yes'  1

Thanks
PSB

**bvdet** · Sep 14 '07, 07:59 PM

Originally posted by psbasha

Hi BV,

I have tried with above piece of code for reading some more filed formats as mentioned below ,the peice of code is not supporting this field format.Can you please suggest how to group for digits and alphanumeric values for the below scenarios.

Code:

Sample.txt
Sample.txt
 
$$$$Header$$$$$$$$$$$$
$$$$Parameter$$$$$$$$$
 
/Parameter_Value/ 1.0
 
/Point/ 
10.0 10.0 10.0 $ Comment: Point Data
20.0 20.0 20.0 
 
$$$$$$Line$$$$$$$$
 
/Line/ $Line Data
 
10.0 15.0 0.0
20.0 10.0 0.0 
 
$$$$$$Circle$$$$$$$$
/Circle/
 
10.0 $Radius
 
0.0 0.0 0.0  $Center


/DashedLineType/			21 $Dashed Line

/XMin_XMax_YMin_YMax/		1 27 1 37 $ Min and Max value

/LineFlag/		yes $ Flag to update


/XY-Plane/ 'Planes'
1,0,0
0,1,0
0,0,0

/XY-Plane/ 'Planes'
2,0,0
0,2,0
0,0,0


/Format/
$Values    
    3     3     1    50    25    28   'Yes'  1

Thanks
PSB

When I write data to a file, I always set up a structured format that is easy to parse. You should try it. This code seems to work:[code=Python]import re

# thanks ilikepython!
def indexList(s, item, start = 0):
return [i + start for (i, obj) in enumerate(s[start:]) if obj == item]

def convertType(s):
for func in (int, float, eval):
try:
n = func(s)
return n
except:
pass
return s

key_patt = re.compile(r'/([A-Za-z_-]+)/')
data_patt = re.compile(r'\d +\.\d+|\d+|\w+' )
fn = 'parameter.txt'

key = None
dd = {}
lineList = [line.strip() for line in open(fn).readli nes() if line != '\n' and not line.startswith ('$')]
for line in lineList:
try:
line = line[:line.index('$' )]
except:
pass
m = key_patt.search (line)
if m:
key = m.group(1)
line1 = line[indexList(line, '/')[1]+1:]
if data_patt.searc h(line1):
if dd.has_key(key) :
dd[key] = dd[key]+[convertType(ite m) for item in data_patt.finda ll(line1)]
else:
dd[key] = [convertType(ite m) for item in data_patt.finda ll(line1)]
else:
dd[key] = []
else:
m1 = data_patt.searc h(line)
if m1:
dd[key].append([convertType(n) for n in data_patt.finda ll(line)])

for key in dd:
print '%s = %s' % (key, dd[key])[/code]

>>> DashedLineType = [21]
Parameter_Value = [1.0]
Point = [[10.0, 10.0, 10.0], [20.0, 20.0, 20.0]]
XY-Plane = ['Planes', [1, 0, 0], [0, 1, 0], [0, 0, 0], 'Planes', [2, 0, 0], [0, 2, 0], [0, 0, 0]]
Format = [[3, 3, 1, 50, 25, 28, 'Yes', 1]]
XMin_XMax_YMin_ YMax = [1, 27, 1, 37]
LineFlag = ['yes']
Line = [[10.0, 15.0, 0.0], [20.0, 10.0, 0.0]]
Circle = [[10.0], [0.0, 0.0, 0.0]]
>>>

**psbasha** · Jan 5 '08, 06:47 AM

Code:

SampleTest
 
$$$$Header$$$$$$$$$$$$
$$$$Parameter$$$$$$$$$
 
/Parameter_range/ 1 1

/Flag1/ 1
/Flag2/ 1
/DummyFlag1/ 1

/STOP/ Line and Circle

$$$$

/LineThick/ 0.1 $$$Line Thickness

$$$$

/Top1/ 10 $$Value1
/Top2/ 11 $$Value2

 $$$
/Bot1/  20 $$Comment
/Bot2/ 30 $$Comment
/Bot4/ 40 $$Comment

$$
/TOl1/ -0.05
/TOl2/ 0.01
 
$$$$$$Line IDs$$$$$$$$
 
/NOT/  10 11 12 1
/NOT/  10 11 12 2
/Ok/   11 12 1  3

/MAT/ $$
1 $Begin
100.	40.	30.	2.0	0 ****22 ksdas
2
200.	40.	60.	2.0	0 ****22 ksdas
3
600.	40.	30.	5.0	0 ****22 ksdas
4
500.	40.	70.	2.0	0 ****22 ksdas
0 $End
2 ***Values $Begin  
1000.  .1
2000.  .2
3000.  .3
4000.  .6
   0.  .0 $End

3 ***Values $Begin  
3000.  .1
5000.  .2
6000.  .3
7000.  .6
   0.  .0 $End
0 $End

2 ***Values $Begin  
1000.  .1
2000.  .2
3000.  .3
4000.  .6
   0.  .0 $End

3 ***Values $Begin  
13000.  .1
45000.  .2
56000.  .3
87000.  .6
    0.  .0 $End
0 $End
2 $Begin
    2.0 .00
    2.0 .210
    3.0 .235
    0.  .0 $End
3 $Begin
    2.0 .00
    2.0 .210
    3.0 .235
    0.  .0 $End
0 $End
/4*ALL/ $ ***
 11       1       1       1     69716.   1000
 11       1       1       5     76296.   1000
 31       1       1       6     74926.   1000
 31       1       1       7     74653.   1000

I have using the above sameple code for reading and storing the data.But I am getting the following error as mentioned below.How to cutomize the above piece of code for reading the above sample file?

PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32.
File "C:\Sample-Mat.py", line 42, in ?
dd[key].append([convertType(n) for n in data_patt.finda ll(line)])

Thanks
PSB

**bvdet** · Jan 5 '08, 03:12 PM

You will have to explain how you need the data tabulated. I have no idea what most of the data is.

**psbasha** · Jan 5 '08, 05:03 PM

Originally posted by bvdet

You will have to explain how you need the data tabulated. I have no idea what most of the data is.

Code:

Description
Hi BV,

we have the kewords in the '/ /'.The respective data is available beside or below the keywords for some cases.

The data should be stored as shwon below ,but using the dict and list using regular expression.

parameter_range = [1,1]

Flag1 = 1
.....

STOP = 'Line and Circel'

Top1 = 10
Top2 = 11
...

Bot1 = 20
Bot2 = 30
....

Tol1 = -0.05
Tol2 = 0.01

NOT = [[ 10,11,12,1],[10,11,12,2]]
OK = [[ 11,12,1,3]]
MAT = { 1:[100.,40.,30.,20.,0],2:[200.,40.,60.,2.0,0],3:[600,40.,30.,5.0,0],4:[500.,40.,70.,2.0,0]}

# 2-integer number  is the start for the block and '0. .0' is the end
MATc = {2:[[1000., 0.1],[2000. ,.2],[3000,0.3],[4000.,0.6]],3:[ [3000.0,0.1],[5000.,0.2],[6000.,.4],[7000.,.6]}
# 0. .0 is the end of the sub block
# o is the end of the block

#Similarly for the other block
# 2-integer number  is the start for the block and '0. .0' is the end
MATT = {2:[[1000., 0.1],[2000. ,.5],[3000,0.3],[4000.,0.6]],3:[ [3000.0,0.9],[5000.,0.2],[6000.,.4],[7000.,.6]}
# 0. .0 is the end of the sub block
# o is the end of the block

#Similarly for the other block
# 2-integer number  is the start for the block and '0. .0' is the end

Factor = {2:[[0.00,2.0],[.210,2.0],[0.235,3.0]],3:[[0.00,2.0],[.2110,2.0],[0.2135,3.0]]}
# 0. .0 is the end of the sub block
# o is the end of the block



ALL = [[ 11,1,1,1,69716.,1000],[ 11,1,1,5,76296.,1000],[ 31,1,1,6,74926.,1000],[ 31, 1,1,7,74653.,1000]]

**bvdet** · Jan 6 '08, 12:58 AM

Where does MATT, MATc, and Factor come from?

**bvdet** · Jan 6 '08, 01:28 AM

BTW, your script fails on your data because all comment lines must begin with '$'. It fails on the first line of data.

**psbasha** · Jan 6 '08, 01:52 AM

Originally posted by bvdet

Where does MATT, MATc, and Factor come from?

Sorry I am explaining how the data can be stored in the dictonary variables or over all data..Its only example to store the data.

MATT,MATc and Factor are variables.

Thanks
PSB

**psbasha** · Jan 6 '08, 01:53 AM

Originally posted by bvdet

BTW, your script fails on your data because all comment lines must begin with '$'. It fails on the first line of data.

Sorry ,all the commnets start with '$' sign.

**bvdet** · Jan 6 '08, 01:53 AM

I made a few minor changes to the code in the earlier solution. Following is the entire source code and output from your data file (with the first line commented out):[code=Python]import re

def indexList(s, item, i=0):
i_list = []
while True:
try:
i = s.index(item, i)
i_list.append(i )
i += 1
except:
break
return i_list

def convertType(s):
for func in (int, float, eval):
try:
n = func(s)
return n
except:
pass
return s

key_patt = re.compile(r'/([A-Za-z_\-0-9]+)/')
data_patt = re.compile(r'\d +\.\d+|\d+|\w+' )

# function to strip comments
def strip_comments( s):
if '$' in s:
return s[:s.index('$')]
elif '*' in s:
return s[:s.index('*')]
return s

def parse_data(fn):
key = None
dd = {}
lineList = [strip_comments( line.strip()) for line in open(fn).readli nes()\
if line != '\n' and not line.startswith ('$')]
for line in lineList:
m = key_patt.search (line)
if m:
key = m.group(1)
line1 = line[indexList(line, '/')[1]+1:]
if data_patt.searc h(line1):
if dd.has_key(key) :
dd[key] = dd[key]+[convertType(ite m) for item in \
data_patt.finda ll(line1)]
else:
dd[key] = [convertType(ite m) for item in \
data_patt.finda ll(line1)]
else:
dd[key] = []
else:
m1 = data_patt.searc h(line)
if m1:
dd[key].append([convertType(n) for n in \
data_patt.finda ll(line)])
return dd

if __name__ == '__main__':
#fn = r'H:\TEMP\temsy s\parameter.txt '
fn = r'H:\TEMP\temsy s\sample_data1. txt'
dataDict = parse_data(fn)
for key in dataDict:
print '%s = %s' % (key, dataDict[key])

>>> Ok = [11, 12, 1, 3]
MAT = [[1], [100, 40, 30, 2.0, 0], [2], [200, 40, 60, 2.0, 0], [3], [600, 40, 30, 5.0, 0], [4], [500, 40, 70, 2.0, 0], [0], [2, 'Values'], [1000, 1], [2000, 2], [3000, 3], [4000, 6], [0, 0], [3, 'Values'], [3000, 1], [5000, 2], [6000, 3], [7000, 6], [0, 0], [0], [2, 'Values'], [1000, 1], [2000, 2], [3000, 3], [4000, 6], [0, 0], [3, 'Values'], [13000, 1], [45000, 2], [56000, 3], [87000, 6], [0, 0], [0], [2], [2.0, 0], [2.0, 210], [3.0, 235], [0, 0], [3], [2.0, 0], [2.0, 210], [3.0, 235], [0, 0], [0], [4, 'ALL'], [11, 1, 1, 1, 69716, 1000], [11, 1, 1, 5, 76296, 1000], [31, 1, 1, 6, 74926, 1000], [31, 1, 1, 7, 74653, 1000]]
LineThick = [0.1000000000000 0001]
TOl2 = [0.01]
TOl1 = [0.0500000000000 00003]
STOP = ['Line', 'and', 'Circle']
Top2 = [11]
Top1 = [10]
Bot4 = [40]
Bot1 = [20]
NOT = [10, 11, 12, 1, 10, 11, 12, 2]
Flag2 = [1]
Flag1 = [1]
Parameter_range = [1, 1]
Bot2 = [30]
DummyFlag1 = [1]
>>> [/code]I understand that this is not your final solution. Maybe you can come up with a way to parse the 'MAT' data.