need help doing a egrep and replace

**ndedhia1** · Mar 19 '09, 02:37 PM

is the $IP_FILE the name of the file that I am trying to print from:
so for me, the $IP_FILE should be: $dbDir/alarmNotificati on.log so should my call looks like this?
[code=unix]
/infra/bin/gawk -f $workDir/grepLogDebug.aw k > $dbDir/alarmNotificati on.log
[/code]

and then inside my grepLogDebug.aw k file
is my print line correct because I am still getting errors:
[code=unix]
BEGIN{
RS="";
FS="low"
}
{
printf "low debug.*\".*\"";
}
END{
[/code]

thanks

**gpraghuram** · Mar 24 '09, 01:13 AM

printf is not ok.
printf in awk is similar to C syntax.

Code:

printf("low debug.*\".*\"");

Raghu

**ndedhia1** · Mar 24 '09, 01:22 PM

I have changed the printf statement and now I get these errors when i try to create the tables:
[code=data]
unable to open file java.io.FileNot FoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-24/alarmNotificati onTmp.txt (No such file or directory)

fatal: error reading input file `stdin': I/O error
[/code]

**gpraghuram** · Mar 25 '09, 12:34 AM

Hi,
I think this discissuion is going on for a long time.
Why cant u post the code and the way u run your script and i can check it.

Raghu

**ndedhia1** · Mar 25 '09, 01:21 PM

Hello Raghu,
Here is the code that I am running:
getAndArchiveIc sDbTest
[code=unix]
#!/bin/ksh
#/infra/sbt-support/supportTools/hsqldb/ics/setProperties

export javaDir=/usr/jdk/jdk1.5.0_06/bin
export icsHost=deepfrz d
export icsLogin=infra
export workDir=/infra/sbt-support/supportTools/hsqldb/ics
export tmpDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
export AppRoot=/infra/sbt-support/supportTools/hsqldb/ics/DBConnection
export hypDB=ICS_DB
export archiveDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
eval $(/infra/bin/EffectiveDate)
export dbDir="$tmpDir/$WebDate"

remoteDbDir="/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
remoteAlarmDir= "/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
previousDatesDi r="/infra/sbt-support/supportTools/hsqldb/ics/archive"

mkdir $dbDir

$workDir/getIcsDbFiles.k sh $remoteDbDir $remoteAlarmDir

/infra/bin/gawk -f $workDir/grepLogDebug.aw k > $dbDir/alarmNotificati on.log | sed -e 's/low.* *"$.*$".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables. awk dbDir=$dbDir

#egrep "low debug.*\".*\"" $dbDir/alarmNotificati on.log | sed -e 's/low.* *"$.*$".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables. awk dbDir=$dbDir

$AppRoot/bin/runConvertMilli s $dbDir

$AppRoot/bin/runDBConnection Hyp $AppRoot/etc/hypProperties $AppRoot/sql/createTextTable s.sql $tmpDir/$WebDate/$hypDB

cd $tmpDir
tar -cf $WebDate.tar $WebDate
gzip -f $WebDate.tar

rm $WebDate.tar.gz
mv $WebDate $1
tar -cvf $1.tar $1
gzip $1.tar
mv $1.tar.gz $previousDatesD ir

# mv -f $WebDate.tar.gz $archiveDir
# rm -rf $WebDate
# rmdir $WebDate
[/code]

grepLogDebug.aw k
[code=unix]
BEGIN{
RS="";
FS="low"
}
{
printf("low debug.*\".*\"");
}
END{
}
[/code]

here is a sample of the alarmNotificati on.log
[code=data]
low debug 2009/3/9 10:33:42.03 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A7089^A 1236612822030^A 23906^A23904^A2 ^Am[0]=23902^A<FINE> <org.apache.act ivemq.broker.re gion.TopicSubsc ription> < Mon 2009/03/09 10:33:38:211 > <org.apache.act ivemq.broker.re gion.TopicSubsc ription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770 ,4,main]> Discarding state cleared, delta-discarded(2819) . TopicSubscripti on: consumer=ID:mdg c01a-47174-1236574999518-0:126:1:2, destination=top ic:///ProdRecap/IDL:consumers/RecapConsumer:1 .0---RecapLocalMD01---local, destinations=1, dispatchedQueue =1970, delivered=41296 526, matched=0, discarded=24533 75^AProdAMQBrok ermdgc01a.out^A ProdLogWatcherm dgc01a^A" 0
low debug 2009/3/9 10:33:46.74 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A7090^A 1236612826740^A 23906^A23904^A2 ^Am[0]=23902^A<FINE> <org.apache.act ivemq.broker.re gion.TopicSubsc ription> < Mon 2009/03/09 10:33:44:486 > <org.apache.act ivemq.broker.re gion.TopicSubsc ription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770 ,4,main]> Discarding state cleared, delta-discarded(16). TopicSubscripti on: consumer=ID:mdg c01a-47174-1236574999518-0:126:1:2, destination=top ic:///ProdRecap/IDL:consumers/RecapConsumer:1 .0---RecapLocalMD01---local, destinations=1, dispatchedQueue =1996, delivered=41322 105, matched=0, discarded=24533 91^AProdAMQBrok ermdgc01a.out^A ProdLogWatcherm dgc01a^A" 0
low debug 2009/3/9 10:33:46.86 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A7091^A 1236612826860^A 19971^A19968^A3 ^Am[0]=19833^A160^APr odBC91x1HybridT radeServer1prdb c91b^AMarketDat aQueue_11^Am[1]=19831^A160^APr odBC91x1HybridT radeServer1prdb c91b^AMarketDat aQueue_11^A" 0
low debug 2009/3/9 10:34:05.22 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A7094^A 1236612845220^A 15601^A13503^A1 ^Am[0]=13501^Acom.cbo e.infrastructur eServices.persi stenceService.U niqueConstraint Exception,No exception message given,Delegatin g stack trace to the causing exception.
java.sql.SQLExc eption: ORA-00001: unique constraint (TRDSESS_10_OWN ER.SBTORDERUK) violated

at oracle.jdbc.dri ver.DatabaseErr or.throwSqlExce ption(DatabaseE rror.java:112)
at oracle.jdbc.dri ver.T4CTTIoer.p rocessError(T4C TTIoer.java:331 )
at oracle.jdbc.dri ver.T4CTTIoer.p rocessError(T4C TTIoer.java:288 )
at oracle.jdbc.dri ver.T4C8Oall.re ceive(T4C8Oall. java:743)
at oracle.jdbc.dri ver.T4CPrepared Statement.doOal l8(T4CPreparedS tatement.java:2 16)
at oracle.jdbc.dri ver.T4CPrepared Statement.execu teForRows(T4CPr eparedStatement .java:955)
at oracle.jdbc.dri ver.OracleState ment.doExecuteW ithTimeout(Orac leStatement.jav a:1167)
at oracle.jdbc.dri ver.OraclePrepa redStatement.se ndBatch(OracleP reparedStatemen t.java:3628)
at oracle.jdbc.dri ver.PhysicalCon nection.commit( PhysicalConnect ion.java:1119)
at com.objectwave. persist.RDBConn ection.commit(R DBConnection.ja va:536)
at com.objectwave. persist.RDBBrok er.commit(RDBBr oker.java:132)
at com.objectwave. persist.BrokerC hangeList.commi tAll(BrokerChan geList.java:59)
at com.objectwave. persist.BrokerT ransactionLog.c ommit(BrokerTra nsactionLog.jav a:88)
at com.objectwave. persist.BrokerT ransactionLog.a ctualCommit(Bro kerTransactionL og.java:53)
at com.objectwave. transactionalSu pport.Transacti onLog.commit(Tr ansactionLog.ja va:298)
at com.objectwave. transactionalSu pport.Session.c ommit(Session.j ava:51)
at com.cboe.infras tructureService s.foundationFra mework.transact ionManagement.B OSession.commit (BOSession.java :99)
at com.cboe.infras tructureService s.foundationFra mework.utilitie s.Transaction.c ommit(Transacti on.java:139)
at com.cboe.ohs.do main.OrdersQueu eReader.process PersistOrderLis t(OrdersQueueRe ader.java:149)
at com.cboe.ohs.do main.OrdersQueu eReader.addTran sactionalBounda ry(OrdersQueueR eader.java:130)
at com.cboe.ohs.do main.OrdersQueu eReader.dequeue AndaddTransacti onBoundaries(Or dersQueueReader .java:63)
at com.cboe.ohs.do main.OrdersQueu eReader.run(Ord ersQueueReader. java:42)
at java.lang.Threa d.run(Thread.ja va:619)
,^AProdBC10x1OH ServerHybridprd bc10a^AOHServer Hybrid^A" 0
[/code]

I call the getAndArchiveIc sDbTest on the command line with a date as an argument..for example:
getAndArchiveIc sDbTest 2009-03-09

That is when i get the errors that I posted earlier.

If I uncomment out the egrep statement in the getAndArchiveIc sDbTest file and comment out the /infra/bin/gawk statement, it runs with no errors, but it does not print out the low debug statements that have multiple lines, from the alarmNotificati on.log file, which is what I am trying to fix.

**gpraghuram** · Mar 26 '09, 07:16 AM

I think problem is this line

Code:

#/infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir 
#change the above line to this

/infra/bin/gawk -f $workDir/grepLogDebug.awk < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir

Raghu

**ndedhia1** · Mar 26 '09, 02:10 PM

The code change got rid of the fatal error:
[code=data]
fatal: error reading input file `stdin': I/O error
[/code]

but I still get this exception error:
[code=data]
unable to open file java.io.FileNot FoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-26/alarmNotificati onTmp.txt (No such file or directory)
[/code]

I do not get this error when I comment out the
/infra/bin/gawk statement and uncomment the egrep statement.

Thanks

**ndedhia1** · Mar 26 '09, 02:57 PM

Another error that I found is that this line:
[code=unix]
/infra/bin/gawk -f $workDir/grepLogDebug.aw k < $dbDir/alarmNotificati on.log
[/code]
creates the output of this, which is incorrect:
[code=data]
low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"
[/code]

The line which I was using before:
[code=unix]
egrep "low debug.*\".*\"" $dbDir/alarmNotificati on.log
[/code]
creates this output which is correct except for the issue I am trying to fix:
[code=data]
low debug 2009/3/9 0:41:40.61 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A792^A1 236577300600^A1 5806^A15402^A1^ Am[0]=15401^A1^AProd fixcas08v2fix8^ AALL^A" 0
low debug 2009/3/9 0:41:40.61 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A795^A1 236577300600^A1 5404^A15402^A1^ Am[0]=15401^A1^AProd fixcas08v2fix8^ AALL^A" 0
low debug 2009/3/9 0:41:40.61 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A796^A1 236577300600^A1 5806^A15402^A1^ Am[0]=15401^A1^AProd fixcas15v2fix1b ^AALL^A" 0
low debug 2009/3/9 0:41:40.61 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A797^A1 236577300600^A1 5404^A15402^A1^ Am[0]=15401^A1^AProd fixcas15v2fix1b ^AALL^A" 0
low debug 2009/3/9 0:41:40.61 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A794^A1 236577300600^A1 1329^A11327^A1^ Am[0]=11326^A1^AProd fixcas08v2fix8^ AALL^A" 0
low debug 2009/3/9 0:41:40.61 ICSNotification Alarm Prodics01ics000 3 IC "1.0^AB^A793^A1 236577300600^A1 0704^A10702^A1^ Am[0]=10701^A1^AProd fixcas15v2fix1b ^AALL^A" 0
[/code]

thanks

**gpraghuram** · Mar 27 '09, 12:01 AM

In the awk you are only combinig the multiple lines and u are not processing anything?
Also u have the replace logic still in egrep?

Raghu

**ndedhia1** · Mar 27 '09, 03:48 PM

are you talking about processing the data in the grepLogDebug.aw k file?

i tried the egrep statement in the grepLogDebug.aw k file:
[code=unix]
FS="low"
}
{
printf("low debug.*\".*\"") ;
egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotificati on.log;
}
END{
}
[/code]
and I call the statement like this in the getAndArchiveIc sDbTest file:
[code=unix]
/infra/bin/gawk -f $workDir/grepLogDebug.aw k $WebDate < $dbDir/alarmNotificati on.log | sed -e 's/low.* *"$.*$".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables. awk dbDir=$dbDir
[/code]
but I get this error:
[code=data]
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.aw k:7: egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotificati on.log;
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.aw k:7: ^ parse error
unable to open file java.io.FileNot FoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-27/alarmNotificati onTmp.txt (No such file or directory)
[/code]

I am not sure where I am supposed to process anything and what I should do with the replace logic that is in the egrep. I though that my grepLogDebug.aw k file would make it so that I would be searching over multiple lines and then the egrep:
[code=unix]
egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotificati on.log;
[/code]
would be able to get the info that I need and sed commands:
[code=unix]
/infra/bin/gawk -f $workDir/grepLogDebug.aw k $WebDate < $dbDir/alarmNotificati on.log | sed -e 's/low.* *"$.*$".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables. awk dbDir=$dbDir
[/code]
would do the replacing and work that I need done on the data that is captured from the grepLogDebug.aw k file.

I am trying to figure out where I am going wrong because I have the logic correct for searching and replacing if all my data is within the same line. I didnt think that because the data is scattered over multiple lines, I would have this much trouble.

**coaxfiber** · Mar 28 '09, 03:30 AM

hi all.,

it's been a long time since i replied/post here. while having some trouble right now doing my script using solaris, i'm getting error:

"grep -hblcnsviw" pattern error

the usual grep is not working here.i hope you could me.

thanks.

coaxfiber

**coaxfiber** · Mar 28 '09, 03:41 AM

coaxfiber> grep -E -e -hblcnsviw 'MTP3b' Health_1_Mar26_ 10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .

coaxfiber> grep -E -e 'MTP3b' -e 'Rnsap' Health_1_Mar26_ 10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .

**gpraghuram** · Mar 29 '09, 11:52 PM

HI,
Why cant u start a new thread instead of appending to an existing thread?

Raghu

**gpraghuram** · Mar 29 '09, 11:53 PM

Hi,
If u want to call egrep from awk then u shuld use system command.
If u want to replace something inside awk then u shuld use sub or gsub.
Better get a awk manual to read abt it.

Raghu

**coaxfiber** · Apr 1 '09, 10:58 AM

hi, can you help me out why i'm not allowed to post new thread.. thanks..

need help doing a egrep and replace

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment