need help doing a egrep and replace

[ndedhia1]

Right now, I am using this line of code to get rid of data in a file called alarmNotificati on that I do not want:

[code=unix]
egrep "low debug.*\".*\"" $dbDir/alarmNotificati on.log
[/code]

but I am having some problem. An entry like the one below in the alarmNotificati on.log file is getting deleted when I only want a part of the entry to be deleted, keeping the rest of the line in tact

[code=data]
low debug 2009/3/9 8:30:21.01 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4600^A 1236605421010^A 15906^A15902^A1 ^Am[0]=15901^Acom.cbo e.exceptions.Da taValidationExc eption,IDL:exce ptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 610712901:SUN:B UY,com.cboe.exc eptions.DataVal idationExceptio n: IDL:exceptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 610712901:SUN:B UY
at com.cboe.util.E xceptionBuilder .dataValidation Exception(Excep tionBuilder.jav a:99)
at com.cboe.busine ssServices.brok erService.Broke rProcessorBase. processUpdate(B rokerProcessorB ase.java:2467)
at com.cboe.busine ssServices.brok erService.Broke rProcessorHybri dImpl.processUp date(BrokerProc essorHybridImpl .java:2388)
at com.cboe.busine ssServices.brok erService.Accep tQuoteUpdateCom mand.doExecute( AcceptQuoteUpda teCommand.java: 109)
at com.cboe.busine ssServices.brok erService.Broke rCommand.execut e(BrokerCommand .java:111)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommandSi ngleThreadImpl. acceptCommands( TradingClassBlo ckCommandSingle ThreadImpl.java :27)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.p rocessBlock(Tra dingClassBlockC ommand.java:203 )
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.e xecute(TradingC lassBlockComman d.java:146)
at com.cboe.server .commandProcess ing.TradingClas sCommandQueuePr ocessor.run(Tra dingClassComman dQueueProcessor .java:51)
at java.lang.Threa d.run(Thread.ja va:619),^AProdBC04x1Hyb ridTradeServer1 prdbc04a^AHybri dTradeServer1^A " 0
[/code]

I want this entry to look like this after I correct the problem:
[code=data]
low debug 2009/3/9 8:30:21.01 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4600^A 1236605421010^A 15906^A15902^A1 ^Am[0]=15901^Acom.cbo e.exceptions.Da taValidationExc eption,IDL:exce ptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 610712901:SUN:B UY,com.cboe.exc eptions.DataVal idationExceptio n: IDL:exceptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 610712901:SUN:B UY^AProdBC04x1H ybridTradeServe r1prdbc04a^AHyb ridTradeServer1 ^A" 0
[/code]


entries like these dont get effected at all by the egrep line:
[code=data]
low debug 2009/3/9 8:30:18.05 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4589^A 1236605418050^A 23605^A23603^A2 ^Am[0]=23601^A60^APro dcas01v2cas0102 ^APOATP/CASQuote^A" 0
low debug 2009/3/9 8:30:19.52 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4590^A 1236605419520^A 23906^A23904^A2 ^Am[0]=23902^A<FINE> <org.apache.act ivemq.broker.re gion.TopicSubsc ription> < Mon 2009/03/09 08:30:15:380 > <org.apache.act ivemq.broker.re gion.TopicSubsc ription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770 ,4,main]> Discarding state cleared, delta-discarded(727). TopicSubscripti on: consumer=ID:mdg c01a-47174-1236574999518-0:126:1:2, destination=top ic:///ProdRecap/IDL:consumers/RecapConsumer:1 .0---RecapLocalMD01---local, destinations=1, dispatchedQueue =1171, delivered=57018 4, matched=0, discarded=742^A ProdAMQBrokermd gc01a.out^AProd LogWatchermdgc0 1a^A" 0
[/code]

Instead of an egrep, is there a global substitute or something that I could do.

thanks

[gpraghuram]

I think u shuld change the regex for this.
As u use . also i thnk it is picking a wrong pattern.
Why cant u use like this
[a-z]a[z][a-z]\.[a-z]a-z]a-z]\. ==> for com.cboe



Raghu

[ndedhia1]

So you mean have the regex like this:
[code=unix]
egrep "low debug.*\"[a-z]a[z][a-z]\.[a-z]a-z]a-z]\.*\"" $dbDir/alarmNotificati on.log
[/code]

is the syntax correct: the a[z] and the a-z]?

thanks

[ndedhia1]

This is the problem actually:
This regex:
[code=unix]
egrep "low debug.*\".*\"" $dbDir/alarmNotificati on.log
[/code]
is looking for data between the two quotation marks:
[code=unix]
".*\"
[/code]

When I hate data like this:
[code=data]
low debug 2009/3/9 8:30:20.47 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4594^A1 236605420470^A1 5906^A15902^A1^ Am[0]=15901^Acom.cbo e.exceptions.Da taValidationExc eption,IDL:exce ptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY,com.cboe.exc eptions.DataVal idationExceptio n: IDL:exceptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:BUY
at com.cboe.util.E xceptionBuilder .dataValidation Exception(Excep tionBuilder.jav a:99)
at com.cboe.busine ssServices.brok erService.Broke rProcessorBase. processUpdate(B rokerProcessorB ase.java:2467)
at com.cboe.busine ssServices.brok erService.Broke rProcessorHybri dImpl.processUp date(BrokerProc essorHybridImpl .java:2388)
at com.cboe.busine ssServices.brok erService.Accep tQuoteUpdateCom mand.doExecute( AcceptQuoteUpda teCommand.java: 109)
at com.cboe.busine ssServices.brok erService.Broke rCommand.execut e(BrokerCommand .java:111)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommandSi ngleThreadImpl. acceptCommands( TradingClassBlo ckCommandSingle ThreadImpl.java :27)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.p rocessBlock(Tra dingClassBlockC ommand.java:203 )
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.e xecute(TradingC lassBlockComman d.java:146)
at com.cboe.server .commandProcess ing.TradingClas sCommandQueuePr ocessor.run(Tra dingClassComman dQueueProcessor .java:51)
at java.lang.Threa d.run(Thread.ja va:619)
,^AProdBC04x1Hy bridTradeServer 1prdbc04a^AHybr idTradeServer1^ A" 0
[/code]
There are line break before each
[code=data]
at com.cboe.util.E xceptionBuilder .dataValidation Exception(Excep tionBuilder.jav a:99)
[/code]
and so there is no closing quote after this line:
[code=data]
low debug 2009/3/9 8:30:20.47 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4594^A1 236605420470^A1 5906^A15902^A1^ Am[0]=15901^Acom.cbo e.exceptions.Da taValidationExc eption,IDL:exce ptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY,com.cboe.exc eptions.DataVal idationExceptio n: IDL:exceptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY
[/code]

I want my output to have everything, including the "at com.cboe..."
[code=data]
low debug 2009/3/9 8:30:20.47 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4593^A1 236605420470^A1 5904^A15902^A1^ Am[0]=15901^Acom.cbo e.exceptions.Da taValidationExc eption,IDL:exce ptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY,com.cboe.exc eptions.DataVal idationExceptio n: IDL:exceptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY at com.cboe.util.E xceptionBuilder .dataValidation Exception(Excep tionBuilder.jav a:99) at com.cboe.busine ssServices.brok erService.Broke rProcessorBase. processUpdate(B rokerProcessorB ase.java:2467) at com.cboe.busine ssServices.brok erService.Broke rProcessorHybri dImpl.processUp date(BrokerProc essorHybridImpl .java:2388) at com.cboe.busine ssServices.brok erService.Accep tQuoteUpdateCom mand.doExecute( AcceptQuoteUpda teCommand.java: 109) at com.cboe.busine ssServices.brok erService.Broke rCommand.execut e(BrokerCommand .java:111) at com.cboe.server .commandProcess ing.TradingClas sBlockCommandSi ngleThreadImpl. acceptCommands( TradingClassBlo ckCommandSingle ThreadImpl.java :27) at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.p rocessBlock(Tra dingClassBlockC ommand.java:203 ) at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.e xecute(TradingC lassBlockComman d.java:146) at com.cboe.server .commandProcess ing.TradingClas sCommandQueuePr ocessor.run(Tra dingClassComman dQueueProcessor .java:51) at java.lang.Threa d.run(Thread.ja va:619),^AProdB C04x1HybridTrad eServer1prdbc04 a^AHybridTradeS erver1^A"
[/code]

but because they are different lines, the egrep doesnt select them. What can I use that will egrep between multiple lines from the beginning to the end of the quotation marks.

This data below works below it searches between the quotation marks and this is all on one line:
[code=data]
low debug 2009/3/9 8:30:19.53 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4592^A1 236605419530^A2 3906^A23904^A2^ Am[0]=23902^A<FINE> <org.apache.act ivemq.broker.re gion.TopicSubsc ription> < Mon 2009/03/09 08:30:18:84 > <org.apache.act ivemq.broker.re gion.TopicSubsc ription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770 ,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscripti on: consumer=ID:mdg c01a-47174-1236574999518-0:126:1:2, destination=top ic:///ProdRecap/IDL:consumers/RecapConsumer:1 .0---RecapLocalMD01---local, destinations=1, dispatchedQueue =890, delivered=59553 1, matched=0, discarded=1542^ AProdAMQBrokerm dgc01a.out^APro dLogWatchermdgc 01a^A" 0
[/code]

thanks

[ndedhia1]

Hi,
Is someone able to help me with this problem

thanks

[ndedhia1]

Is there something that I can use to search in a file over more then 1 line.
I believe that egrep only works line by line but can i search between lines?

thanks

[gpraghuram]

Hi,
Then i think you shuld wite your code using AWK.
I know AWK has this feature.

Raghu

[ndedhia1]

Thanks gpraghuram,

Is there a code snippet you have that would help me

thanks

[gpraghuram]

Hi,
I dont have a example code with me.
But check this link
and look for this "Multiline Records" in the page.

Raghu

[ndedhia1]

I checked out the link and I understand what they are saying:
[code=data]
Multiline Records

By default, the input record separator RS recognizes a newline as the marker between records. As is the norm in awk, this can be changed to allow for multiline records. When RS is set to the null string, the newline character always acts as a field separator, in addition to whatever value FS might have.
[/code]

But how do I change RS so that it looks between two quotation marks:
[code=unix]
".*\"
[/code]
and the awk command that does that.

thanks

[ndedhia1]

I used this snippet of code
[code=unix]
awk -Flow print "low debug.*\".*\"" $dbDir/alarmNotificati on.log
[/code]
I wanted the lines to be separated by the word low like in these entries:
[code=data]
low debug 2009/3/9 8:30:19.53 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4591^A 1236605419530^A 23906^A23904^A2 ^Am[0]=23902^A<FINE> <org.apache.act ivemq.broker.re gion.TopicSubsc ription> < Mon 2009/03/09 08:30:16:775 > <org.apache.act ivemq.broker.re gion.TopicSubsc ription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770 ,4,main]> Discarding state cleared, delta-discarded(18). TopicSubscripti on: consumer=ID:mdg c01a-47174-1236574999518-0:126:1:2, destination=top ic:///ProdRecap/IDL:consumers/RecapConsumer:1 .0---RecapLocalMD01---local, destinations=1, dispatchedQueue =0, delivered=58274 3, matched=0, discarded=760^A ProdAMQBrokermd gc01a.out^AProd LogWatchermdgc0 1a^A" 0
low debug 2009/3/9 8:30:19.53 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4592^A 1236605419530^A 23906^A23904^A2 ^Am[0]=23902^A<FINE> <org.apache.act ivemq.broker.re gion.TopicSubsc ription> < Mon 2009/03/09 08:30:18:84 > <org.apache.act ivemq.broker.re gion.TopicSubsc ription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770 ,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscripti on: consumer=ID:mdg c01a-47174-1236574999518-0:126:1:2, destination=top ic:///ProdRecap/IDL:consumers/RecapConsumer:1 .0---RecapLocalMD01---local, destinations=1, dispatchedQueue =890, delivered=59553 1, matched=0, discarded=1542^ AProdAMQBrokerm dgc01a.out^APro dLogWatchermdgc 01a^A" 0
low debug 2009/3/9 8:30:20.47 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4594^A 1236605420470^A 15906^A15902^A1 ^Am[0]=15901^Acom.cbo e.exceptions.Da taValidationExc eption,IDL:exce ptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY,com.cboe.exc eptions.DataVal idationExceptio n: IDL:exceptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY
at com.cboe.util.E xceptionBuilder .dataValidation Exception(Excep tionBuilder.jav a:99)
at com.cboe.busine ssServices.brok erService.Broke rProcessorBase. processUpdate(B rokerProcessorB ase.java:2467)
at com.cboe.busine ssServices.brok erService.Broke rProcessorHybri dImpl.processUp date(BrokerProc essorHybridImpl .java:2388)
at com.cboe.busine ssServices.brok erService.Accep tQuoteUpdateCom mand.doExecute( AcceptQuoteUpda teCommand.java: 109)
at com.cboe.busine ssServices.brok erService.Broke rCommand.execut e(BrokerCommand .java:111)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommandSi ngleThreadImpl. acceptCommands( TradingClassBlo ckCommandSingle ThreadImpl.java :27)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.p rocessBlock(Tra dingClassBlockC ommand.java:203 )
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.e xecute(TradingC lassBlockComman d.java:146)
at com.cboe.server .commandProcess ing.TradingClas sCommandQueuePr ocessor.run(Tra dingClassComman dQueueProcessor .java:51)
at java.lang.Threa d.run(Thread.ja va:619)
,^AProdBC04x1Hy bridTradeServer 1prdbc04a^AHybr idTradeServer1^ A" 0
low debug 2009/3/9 8:30:20.47 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A4593^A 1236605420470^A 15904^A15902^A1 ^Am[0]=15901^Acom.cbo e.exceptions.Da taValidationExc eption,IDL:exce ptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY,com.cboe.exc eptions.DataVal idationExceptio n: IDL:exceptions/DataValidationE xception:1.0 com.cboe.except ions.OrderBookT radableNotFound Exception: No tradables in book at specified price: 639898851:SUN:B UY
at com.cboe.util.E xceptionBuilder .dataValidation Exception(Excep tionBuilder.jav a:99)
at com.cboe.busine ssServices.brok erService.Broke rProcessorBase. processUpdate(B rokerProcessorB ase.java:2467)
at com.cboe.busine ssServices.brok erService.Broke rProcessorHybri dImpl.processUp date(BrokerProc essorHybridImpl .java:2388)
at com.cboe.busine ssServices.brok erService.Accep tQuoteUpdateCom mand.doExecute( AcceptQuoteUpda teCommand.java: 109)
at com.cboe.busine ssServices.brok erService.Broke rCommand.execut e(BrokerCommand .java:111)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommandSi ngleThreadImpl. acceptCommands( TradingClassBlo ckCommandSingle ThreadImpl.java :27)
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.p rocessBlock(Tra dingClassBlockC ommand.java:203 )
at com.cboe.server .commandProcess ing.TradingClas sBlockCommand.e xecute(TradingC lassBlockComman d.java:146)
at com.cboe.server .commandProcess ing.TradingClas sCommandQueuePr ocessor.run(Tra dingClassComman dQueueProcessor .java:51)
at java.lang.Threa d.run(Thread.ja va:619)
,^AProdBC04x1Hy bridTradeServer 1prdbc04a^AHybr idTradeServer1^ A" 0
[/code]

and I wanted it to print everything in the line:
[code=unix]
low debug.*\".*\"
[/code]

from the file:
[code=unix]
$dbDir/alarmNotificati on.log
[/code]

BUT i get these errors:
[code=data]
awk: syntax error near line 1
awk: bailing out near line 1
unable to open file java.io.FileNot FoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-17/alarmNotificati onTmp.txt (No such file or directory)
[/code]

What am i doing wrong?

thanks

[gpraghuram]

Instead of command line write a awk program which will be easy.

BEGIN{
RS="";
FS="low"
}
{
/*write code here for processing the line*/
}
END{
}

Raghu

[ndedhia1]

I created an AWK file called grepLogDebug.aw k which has this code:
[code=unix]
BEGIN{
RS="";
FS="low"
}
{
egrep "low debug.*\".*\"" $dbDir/alarmNotificati on.log;
}
END{
[/code]

and here is how I call the awk file:
[code=unix]
/infra/bin/gawk -f $workDir/grepLogDebug.aw k
[/code]

and i get this error:
[code=data]
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.aw k:6: egrep "low debug.*\".*\"" $dbDir/alarmNotificati on.log;
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.aw k:6: ^ parse error
unable to open file java.io.FileNot FoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-18/alarmNotificati onTmp.txt (No such file or directory)
[/code]

[ndedhia1]

I changed some of the code:

To call the AWK file:
[code=unix]
/infra/bin/gawk -f $workDir/grepLogDebug.aw k dbDir=$dbDir
[/code]

and in the AWK file:
[code=unix]
BEGIN{
RS="";
FS="low"
}
{
printf "low debug.*\".*\"" > (dbDir "/alarmNotificati on.log");
}
END{
}
[/code]

but when i run it,
it get stuck and doesnt finish creating the tables

thanks

[gpraghuram]

But where u are giving file name .
call like this
/infra/bin/gawk -f $workDir/grepLogDebug.aw k dbDir=$dbDir < $IP_FILE

also you cant directly get the value of dbdir inside the AWK script and there is a special syntax to pass input arguments to the AWK.
Check in gawk manual

Raghu