Buffered Reader

[jyohere]

I am using buffered Reader.....some problem arises ...dnt know why....am using the following code to read a big file about 8MB.

path = "/home/t.txt"

File file=new File(path);
FileReader freader=new FileReader(file );
BufferedReader reader=new BufferedReader( freader);

System.out.prin tln(reader.read y());
while((line = reader.readLine ()) != null)
{
//some code
}

in the above case it sometimes gives reader.ready() is false....and so it does not enter the while loop wat could be the fix....and why its giving reader.ready() is false
reason and wat is the fix for it...

[JosAH]

I am using buffered Reader.....some problem arises ...dnt know why....am using the following code to read a big file about 8MB.

path = "/home/t.txt"

File file=new File(path);
FileReader freader=new FileReader(file );
BufferedReader reader=new BufferedReader( freader);

System.out.prin tln(reader.read y());
while((line = reader.readLine ()) != null)
{
//some code
}

in the above case it sometimes gives reader.ready() is false....and so it does not enter the while loop wat could be the fix....and why its giving reader.ready() is false
reason and wat is the fix for it...

No matter the value of ready(), your code enters that while loop as far as I can see.
Did you show us the code you were running?

I ripped the following straight from the API docs for the BufferedReader class:

Tells whether this stream is ready to be read. A buffered character stream is ready if the buffer is not empty, or if the underlying character stream is ready.

When the reader is freshly opened, it may not have filled its buffer yet; that could
be a reason for ready() to return false. Don't pay any attention to it and just
read away.

kind regards,

Jos

[jyohere]

No matter the value of ready(), your code enters that while loop as far as I can see.
Did you show us the code you were running?

I ripped the following straight from the API docs for the BufferedReader class:



When the reader is freshly opened, it may not have filled its buffer yet; that could
be a reason for ready() to return false. Don't pay any attention to it and just
read away.

kind regards,

Jos

This is the same code am using.....i printed reader.ready() only after it was not entering the loop... i am using it in my project now....i got the error when it was
tested for stress test....it was working fine when the file size was small...

[jyohere]

This is the same code am using.....i printed reader.ready() only after it was not entering the loop... i am using it in my project now....i got the error when it was
tested for stress test....it was working fine when the file size was small...

The code i said above is in a function....it will get called and then the while loop has to execute immediately the first time itself....

[JosAH]

This is the same code am using.....i printed reader.ready() only after it was not entering the loop... i am using it in my project now....i got the error when it was
tested for stress test....it was working fine when the file size was small...

I'm afraid the error is somewhere else then; check if the file exists; if so read it
and see if the while loop *terminates* properly. Maybe an unchecked exception
what thrown somewhere in the body of that while loop.

If you wrap a BufferedReader around a FileReader, the ready() method doesn't
tell you much.

kind regards,

Jos

[jyohere]

I'm afraid the error is somewhere else then; check if the file exists; if so read it
and see if the while loop *terminates* properly. Maybe an unchecked exception
what thrown somewhere in the body of that while loop.

If you wrap a BufferedReader around a FileReader, the ready() method doesn't
tell you much.

kind regards,

Jos

then why is it not entering the loop only when reader.ready() is false?

[JosAH]

then why is it not entering the loop only when reader.ready() is false?

I don't know because you didn't show us any relevant code that showed the
symptoms you keep on mentioning.

kind regards,

Jos

[jyohere]

I don't know because you didn't show us any relevant code that showed the
symptoms you keep on mentioning.

kind regards,

Jos

this is the method.....sry its too big Jos

public static ArrayList get(int ne_id,int ne_type,

int shelf_id, int card_id, int port_id,

int index1_id, int index2_id,

int index3_id,int index4_id, int[] location,

NameAndStringVa lue_T[] name,String[] granularity,

String[] pmLocation, short[] layerRate,

String startTime, String endTime){



String[] pmParameters={" PMP_BBE", "PMP_SES", "PMP_ES", "PMP_UAS"};

int index=0;

logger.info("In side get()");



BufferedReader reader=null;

String line=" ";

ArrayList keyList=new ArrayList();

ArrayList valueList=new ArrayList();

String str[]=null;

String syearStr=" ", eyearStr=" ";

String smonthStr=" ", emonthStr=" ";

String sdateStr=" ", edateStr=" ";

int startDate=0, endDate=0;

StringBuffer fileName=new StringBuffer();

StringBuffer endfileName=new StringBuffer();

String path=" ";

Calendar start_cal,end_c al;

int start_week_of_y ear=0,end_week_ of_year=0;

boolean near=false;



SimpleDateForma t dateFormat=new SimpleDateForma t("yyyyMMddHHmm ss");



String sTime=startTime .substring(0,14 );

String eTime=endTime.s ubstring(0,14);



ParsePosition p1=new ParsePosition(0 );

Date start=dateForma t.parse(sTime,p 1);

ParsePosition p2=new ParsePosition(0 );

Date end=dateFormat. parse(eTime,p2) ;



logger.info(HDR + " START TIME : " + start);

logger.info(HDR + " END TIME : " + end);

try{

Matcher m;

if(granularity[0].equalsIgnoreCa se("15min")){

syearStr=startT ime.substring(0 ,4);



fileName.append (syearStr);

fileName.append ("_");

smonthStr=start Time.substring( 4,6);

fileName.append (smonthStr);

fileName.append ("_");

sdateStr=startT ime.substring(6 ,8);

fileName.append (sdateStr);

fileName.append ("_");

fileName.append ("15M");

fileName.append ("_");

if(ne_type<=9){

fileName.append ("0");

fileName.append (ne_type);

}else{

fileName.append (ne_type);

}

fileName.append (".ppm");



eyearStr=endTim e.substring(0,4 );

emonthStr=endTi me.substring(4, 6);

edateStr=endTim e.substring(6,8 );



startDate=Integ er.parseInt(sda teStr);

endDate=Integer .parseInt(edate Str);



}else if(granularity[0].equalsIgnoreCa se("24H")){



syearStr=startT ime.substring(0 ,4);

int start_year=Inte ger.parseInt(sy earStr);



fileName.append (syearStr);

fileName.append ("_");



smonthStr=start Time.substring( 4,6);

int start_month=Int eger.parseInt(s monthStr)-1;

sdateStr=startT ime.substring(6 ,8);

int start_date=Inte ger.parseInt(sd ateStr);



start_cal=new GregorianCalend ar(start_year,s tart_month,star t_date);

start_cal.setMi nimalDaysInFirs tWeek(7);

start_cal.setFi rstDayOfWeek(Ca lendar.MONDAY);



if ((start_cal.get (Calendar.MONTH ) == Calendar.JANUAR Y) && (start_cal.get( Calendar.WEEK_O F_YEAR) >= 52))

start_week_of_y ear=0;

else

start_week_of_y ear=start_cal.g et(Calendar.WEE K_OF_YEAR);



logger.info(HDR +"start_week_of _year " +start_week_of_ year);



if(start_week_o f_year <=9){

fileName.append ("0");

fileName.append (start_week_of_ year);

}else{

fileName.append (start_week_of_ year);

}

fileName.append ("_");

fileName.append ("24H");

fileName.append ("_");

if(ne_type<=9){

fileName.append ("0");

fileName.append (ne_type);

}else{

fileName.append (ne_type);

}

fileName.append (".ppm");



eyearStr=endTim e.substring(0,4 );

int end_year=Intege r.parseInt(eyea rStr);



emonthStr=endTi me.substring(4, 6);

int end_month=Integ er.parseInt(emo nthStr)-1;

edateStr=endTim e.substring(6,8 );

int end_date=Intege r.parseInt(edat eStr);



end_cal=new GregorianCalend ar(end_year,end _month,end_date );

end_cal.setMini malDaysInFirstW eek(7);

end_cal.setFirs tDayOfWeek(Cale ndar.MONDAY);



if ((end_cal.get(C alendar.MONTH) == Calendar.JANUAR Y) && (end_cal.get(Ca lendar.WEEK_OF_ YEAR) >= 52))

end_week_of_yea r=0;

else

end_week_of_yea r=end_cal.get(C alendar.WEEK_OF _YEAR);



logger.info(HDR +"end_week_of_y ear " +end_week_of_ye ar);



}







path = "/var/ftp/PMFiles/marconi/" + fileName;



logger.info(HDR +"path"+path );



File file=new File(path);

reader=new BufferedReader( new FileReader(file ));

logger.info(HDR +"Before while loop");

while((line = reader.readLine ()) != null)

{



logger.info(HDR +"Inside while loop");

String[] testStr=line.sp lit(REGEX1);



try{

int test=Integer.pa rseInt(testStr[testStr.length-1]);

}catch(Exceptio n e){





String newLine= reader.readLine ();

StringBuffer buffer=new StringBuffer(li ne);

buffer.append(n ewLine);

line=buffer.toS tring();



}



m=pattern.match er(line);



boolean matchFound = m.find();

if (matchFound){

str=new String[m.groupCount()+ 1];



for (int i=0; i<=m.groupCount (); i++)

str[i]=m.group(i);

}



int neId=Integer.pa rseInt(str[2]);

//logger.info(HDR +"neId " +neId);

int neType=Integer. parseInt(str[3]);

//logger.info(HDR +"neType " +neType);

int shelfId=Integer .parseInt(str[4]);

int cardId=Integer. parseInt(str[6]);

int portId=Integer. parseInt(str[9]);

//logger.info(HDR +"ptp" + shelfId+ " " +cardId +" "+ portId);

int indexId1=Intege r.parseInt(str[10]);

int indexId2=Intege r.parseInt(str[11]);

int indexId3=Intege r.parseInt(str[12]);

int indexId4=Intege r.parseInt(str[13]);

//logger.info(HDR +"ctp" + indexId1+ " " +indexId2 +" "+ indexId3+" "+indexId4) ;

int entityType=Inte ger.parseInt(st r[14]);

//logger.info(HDR +"entity " +entityType);



ParsePosition p=new ParsePosition(0 );

Date time=dateFormat .parse(str[18],p);

//logger.info(HDR +"time " +time);



if(((entityType ==location[0]||entityType==l ocation[1]))

&& (time.after(sta rt) ||time.equals(s tart))

&&(time.before( end) ||time.equals(e nd))

&& neId==ne_id && neType==ne_type && shelfId==shelf_ id

&& indexId1==index 1_id && indexId2==index 2_id && indexId3==index 3_id && indexId4==index 4_id

&& cardId==card_id && portId ==port_id){





logger.info("Ma tched so inside data collection...") ;

StringBuffer key=new StringBuffer();



for(int l=1;l<13;l++){



key.append(str[l]);

key.append(":") ;

}



key.append(str[18]);

key.append(".0" );

String keyStr=key.toSt ring();



StringBuffer value=new StringBuffer();



for(int k=0;k<4;k++){

value.append(na me[k].value);

value.append(", ");

}



value.append(gr anularity[0]);

value.append(", ");



value.append(la yerRate[0]);

value.append(", ");



value.append(st r[18]);

value.append(". 0");

value.append(", ");



value.append(st r[14]);

value.append(": ");

if (entityType==lo cation[0]){

value.append(pm Location[0]);

near=true;

}else if (entityType==lo cation[1]){

value.append(pm Location[1]);

near=false;

}

value.append(", ");



String pmMeasurement_1 =null;

String pmMeasurement_2 =null;

String result=null;



if(!keyList.con tains(keyStr)){

pmMeasurement_1 =getPMMeasureme nt(pmParameters ,str,true,near, pmLocation,loca tion);

value.append(pm Measurement_1);



result=value.to String();



}else{

pmMeasurement_2 =getPMMeasureme nt(pmParameters ,str,false,near ,pmLocation,loc ation);

}



if(!keyList.con tains(keyStr)){



keyList.add(ind ex,keyStr);

valueList.add(i ndex,result);

index++;

}else{

int position=keyLis t.indexOf(keySt r);

String temp=(String)va lueList.get(pos ition);

String[] tempArray=temp. split(",");

valueList.remov e(position);

StringBuffer tempBuffer=new StringBuffer();

for(int i=0;i<(tempArra y.length-4);i++){

tempBuffer.appe nd(tempArray[i]);

tempBuffer.appe nd(",");

}



tempBuffer.appe nd(pmMeasuremen t_2);

valueList.add(p osition, tempBuffer.toSt ring());



}







}//if

}//end while





if((startDate != endDate) || (start_week_of_ year != end_week_of_yea r)){



logger.info(HDR +"Start and End Not Equal");



if(granularity[0].equalsIgnoreCa se("15min")){



endfileName.app end(eyearStr);

endfileName.app end("_");

endfileName.app end(emonthStr);

endfileName.app end("_");

endfileName.app end(edateStr);

endfileName.app end("_");

endfileName.app end("15M");

endfileName.app end("_");

if(ne_type<=9){

endfileName.app end("0");

endfileName.app end(ne_type);

}else{

endfileName.app end(ne_type);

}

endfileName.app end(".ppm");



}else if(granularity[0].equalsIgnoreCa se("24H")){



endfileName.app end(eyearStr);

endfileName.app end("_");

if(end_week_of_ year <=9){

endfileName.app end("0");

endfileName.app end(end_week_of _year);

}else{

endfileName.app end(end_week_of _year);

}



endfileName.app end("_");

endfileName.app end("24H");

endfileName.app end("_");

if(ne_type<=9){

endfileName.app end("0");

endfileName.app end(ne_type);

}else{

endfileName.app end(ne_type);

}

endfileName.app end(".ppm");

}



logger.info(HDR +"endfileNam e "+endfileNa me);



path = "/var/ftp/PMFiles/marconi/" + endfileName;



logger.info(HDR +"path "+path);



File endfile=new File(path);

reader=new BufferedReader( new FileReader(endf ile));

while((line = reader.readLine ()) != null)

{



String[] testStr=line.sp lit(REGEX1);



try{

int test=Integer.pa rseInt(testStr[testStr.length-1]);

}catch(Exceptio n e){



String newLine= reader.readLine ();



StringBuffer buffer=new StringBuffer(li ne);

buffer.append(n ewLine);

line=buffer.toS tring();



}





m=pattern.match er(line);



boolean matchFound = m.find();

if (matchFound){

str=new String[m.groupCount()+ 1];

for (int i=0; i<=m.groupCount (); i++)

str[i]=m.group(i);

}



int neId=Integer.pa rseInt(str[2]);

int neType=Integer. parseInt(str[3]);

int shelfId=Integer .parseInt(str[4]);

int cardId=Integer. parseInt(str[6]);

int portId=Integer. parseInt(str[9]);

int indexId1=Intege r.parseInt(str[10]);

int indexId2=Intege r.parseInt(str[11]);

int indexId3=Intege r.parseInt(str[12]);

int indexId4=Intege r.parseInt(str[13]);

int entityType=Inte ger.parseInt(st r[14]);

ParsePosition p=new ParsePosition(0 );

Date time=dateFormat .parse(str[18],p);



if(((entityType ==location[0]||entityType==l ocation[1]))

&&(time.before( end) ||time.equals(e nd))

&& neId==ne_id && neType==ne_type && shelfId==shelf_ id

&& indexId1==index 1_id && indexId2==index 2_id && indexId3==index 3_id && indexId4==index 4_id

&& cardId==card_id && portId ==port_id){





StringBuffer key=new StringBuffer();



for(int l=1;l<13;l++){



key.append(str[l]);

key.append(":") ;

}



key.append(str[18]);

key.append(".0" );



String keyStr=key.toSt ring();



StringBuffer value=new StringBuffer();



for(int k=0;k<4;k++){

value.append(na me[k].value);

value.append(", ");

}



value.append(gr anularity[0]);

value.append(", ");



value.append(la yerRate[0]);

value.append(", ");



value.append(st r[18]);

value.append(". 0");

value.append(", ");



value.append(st r[14]);

value.append(": ");

if (entityType==lo cation[0]){

value.append(pm Location[0]);

near=true;

}else if (entityType==lo cation[1]){

value.append(pm Location[1]);

near=false;

}

value.append(", ");



String pmMeasurement_1 =null;

String pmMeasurement_2 =null;

String result=null;



if(!keyList.con tains(keyStr)){

pmMeasurement_1 =getPMMeasureme nt(pmParameters ,str,true,near, pmLocation,loca tion);

value.append(pm Measurement_1);



result=value.to String();



}else{

pmMeasurement_2 =getPMMeasureme nt(pmParameters ,str,false,near ,pmLocation,loc ation);

}



if(!keyList.con tains(keyStr)){



keyList.add(ind ex,keyStr);

valueList.add(i ndex,result);

index++;

}else{

int position=keyLis t.indexOf(keySt r);

String temp=(String)va lueList.get(pos ition);

String[] tempArray=temp. split(",");

valueList.remov e(position);

StringBuffer tempBuffer=new StringBuffer();

for(int i=0;i<(tempArra y.length-4);i++){

tempBuffer.appe nd(tempArray[i]);

tempBuffer.appe nd(",");

}

tempBuffer.appe nd(pmMeasuremen t_2);

valueList.add(p osition, tempBuffer.toSt ring());



}



}

}



}



}catch(FileNotF oundException e){

logger.warn(HDR +e.getMessage() );

logger.warn("", e);

}catch(IndexOut OfBoundsExcepti on e){

logger.warn(HDR +e.getMessage() );

logger.warn("", e);

}catch(PatternS yntaxException e){

logger.warn(HDR +e.getMessage() );

logger.warn("", e);

}catch(IOExcept ion e){

logger.warn(HDR +e.getMessage() );

logger.warn("", e);

}catch(NullPoin terException e){

logger.warn(HDR +e.getMessage() );

logger.warn("", e);

}catch(Exceptio n e){

logger.warn(HDR +e.getMessage() );

logger.warn("", e);

}finally{

try{

keyList=null;

if (reader!=null)

reader.close();

}catch(IOExcept ion e){

logger.warn(HDR +e.getMessage() );

logger.warn("", e);

}

}



for(int k=0;k<valueList .size();k++)

logger.info("Fi nal valueList Returned is " +valueList.get( k));

logger.info(HDR +"Returned valueList ");

return valueList;



}

[JosAH]

this is the method.....sry its too big Jos

It sure is ;-) What ends up in the log output? Does the log output flush itself
after a single line is written to it?

kind regards,

Jos

[jyohere]

It sure is ;-) What ends up in the log output? Does the log output flush itself
after a single line is written to it?

kind regards,

Jos

The while loop is not executing only sometimes....it does not enter inside....when i
type reader.ready(). ...it gives false before the loop....wat could the problem ....can it be because of too much load in the server.....one more thing is that for every 15 minutes the file gets overwritten.... .but if the file is not there it should have thrown file not found exception isn't it....pls help....have been spending time on this for quite a long time....give me the possible chances of error...

[JosAH]

The while loop is not executing only sometimes....it does not enter inside....when i
type reader.ready(). ...it gives false before the loop....wat could the problem ....can it be because of too much load in the server.....one more thing is that for every 15 minutes the file gets overwritten.... .but if the file is not there it should have thrown file not found exception isn't it....pls help....have been spending time on this for quite a long time....give me the possible chances of error...

So another (external) process is fiddling with that file. I'm beginning to understand
while that entire while loop is skipped: suppose you open the file successfully,
then the file content is changed and your process can't read the (previous)
content anymore because it hadn't started reading yet.

Either make that other process lock the overwriting process; by using an (empty)
lock file on which both processes negoatiate or (this is hacking) make your
process try to reopen the entire thing if opening the file succeeded but the first
read did not.

Your two processes should use a locking mechanism though otherwise you're
asking for trouble (which you already experienced ;-)

kind regards,

Jos

[jyohere]

So another (external) process is fiddling with that file. I'm beginning to understand
while that entire while loop is skipped: suppose you open the file successfully,
then the file content is changed and your process can't read the (previous)
content anymore because it hadn't started reading yet.

Either make that other process lock the overwriting process; by using an (empty)
lock file on which both processes negoatiate or (this is hacking) make your
process try to reopen the entire thing if opening the file succeeded but the first
read did not.

Your two processes should use a locking mechanism though otherwise you're
asking for trouble (which you already experienced ;-)

kind regards,

Jos

the other process is a crontab entry in linux which will fetch the file from an other system and overwrite it.....i am asking about the load because when the file is of small size in the beginning it is working fine...but when the file size increases to say 3mb it is behaving differently.... hope its clear for u now..

[JosAH]

the other process is a crontab entry in linux which will fetch the file from an other system and overwrite it.....i am asking about the load because when the file is of small size in the beginning it is working fine...but when the file size increases to say 3mb it is behaving differently.... hope its clear for u now..

If the other process doesn't want to play that locking game all bets are off. You
could check (before you start reading) whether or not the file size is changing.
If so, don't read because the other process is writing. If it doesn't change, hope
for the best and start reading.

kind regards,

Jos

[jyohere]

If the other process doesn't want to play that locking game all bets are off. You
could check (before you start reading) whether or not the file size is changing.
If so, don't read because the other process is writing. If it doesn't change, hope
for the best and start reading.

kind regards,

Jos

I am testing the file putting it in a separate place.....let me try my luck

[JosAH]

I am testing the file putting it in a separate place.....let me try my luck

I wish you all the best of luck.

kind regards,

Jos