parsing website with a "searching...." page

[Aaron]

I'm trying to parse a table on a webpage to pull down some data I
need. The page is based off of information entered into a form. when
you submit the data from the form it displays a "Searching. .." page
then, refreshes and displays the table I want. I have code that grabs
data from the page using cURL but when I look at the data it contains
the "Searching. .." page and not the table that I want. below is the
code i have so far....Thanks in advance for any help.

<?php

$url="http://www.website.com ";

$post_data = array();
$post_data['postvar1'] = "val1";
$post_data['postvar2'] = "val2";


$o="";
foreach($post_d ata as $k=>$v)
{
$o.= "$k=".utf8_enco de($v)."&";
}

$post_data=subs tr($o,0,-1);


$ch= curl_init();
curl_setopt($ch , CURLOPT_POST,1) ;
curl_setopt($ch , CURLOPT_HEADER, 0);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
curl_setopt($ch , CURLOPT_URL,$ur l);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $post_data);
$result = curl_exec($ch);
curl_close($ch) ;
$result=explode ("\n",$resul t);

?>

[Erwin Moller]

Re: parsing website with a &quot;searching ....&quot; page

Aaron wrote:

I'm trying to parse a table on a webpage to pull down some data I
need. The page is based off of information entered into a form. when
you submit the data from the form it displays a "Searching. .." page
then, refreshes and displays the table I want. I have code that grabs
data from the page using cURL but when I look at the data it contains
the "Searching. .." page and not the table that I want. below is the
code i have so far....Thanks in advance for any help.
>
<?php
>
$url="http://www.website.com ";
>
$post_data = array();
$post_data['postvar1'] = "val1";
$post_data['postvar2'] = "val2";
>
>
$o="";
foreach($post_d ata as $k=>$v)
{
$o.= "$k=".utf8_enco de($v)."&";
}
>
$post_data=subs tr($o,0,-1);
>
>
$ch= curl_init();
curl_setopt($ch , CURLOPT_POST,1) ;
curl_setopt($ch , CURLOPT_HEADER, 0);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
curl_setopt($ch , CURLOPT_URL,$ur l);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $post_data);
$result = curl_exec($ch);
curl_close($ch) ;
$result=explode ("\n",$resul t);
>
?>

Hi,

The page is probably using JavaScript to give that effect.
Inspect the content of $result to see if this is the case.
Look for divs that are not visible and made visible when the page loads (end
of script, or an onLoad event).
The data you want might very well be inside the $result.

If not, give more information WHAT the $result contained.

Regards,
Erwin Moller

[Aaron]

Re: parsing website with a &quot;searching ....&quot; page

On Apr 6, 10:55 am, Erwin Moller
<since_humans_r ead_this_I_am_s pammed_too_m... @spamyourself.c omwrote:

Aaron wrote:

I'm trying to parse a table on a webpage to pull down some data I
need. The page is based off of information entered into a form. when
you submit the data from the form it displays a "Searching. .." page
then, refreshes and displays the table I want. I have code that grabs
data from the page using cURL but when I look at the data it contains
the "Searching. .." page and not the table that I want. below is the
code i have so far....Thanks in advance for any help.

>

<?php

>

$url="http://www.website.com ";

>

$post_data = array();
$post_data['postvar1'] = "val1";
$post_data['postvar2'] = "val2";

>

$o="";
foreach($post_d ata as $k=>$v)
{
$o.= "$k=".utf8_enco de($v)."&";
}

>

$post_data=subs tr($o,0,-1);

>

$ch= curl_init();
curl_setopt($ch , CURLOPT_POST,1) ;
curl_setopt($ch , CURLOPT_HEADER, 0);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
curl_setopt($ch , CURLOPT_URL,$ur l);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $post_data);
$result = curl_exec($ch);
curl_close($ch) ;
$result=explode ("\n",$resul t);

>

?>

>
Hi,
>
The page is probably using JavaScript to give that effect.
Inspect the content of $result to see if this is the case.
Look for divs that are not visible and made visible when the page loads (end
of script, or an onLoad event).
The data you want might very well be inside the $result.
>
If not, give more information WHAT the $result contained.
>
Regards,
Erwin Moller- Hide quoted text -
>
- Show quoted text -

Heres basicly what was returned by $result

<!-- Vignette V6 Fri Apr 06 09:56:47 2007 -->


<script language="javas cript">

function changePassword( ) {

window.open('ht tps://www.secure-hallmark.xxxxx. com/ldap/
em_change_passw ord.asp', 'ChangePassword ',
'height=250,wid th=400,screenX= 100,screenY=100 ');
}

function forgotPassword( ) {

window.open('/auth/forgotpasswordf orm/0,2290,,00.html ',
'ForgotPassword ', 'height=250,wid th=400,screenX= 100,screenY=100 ');
}
</script><html><h ead><title>xxxx xxx</title>
<!--INCLUDE VIRTUAL="/ast/functions.asp" -->

<script language="Javas cript">
//Code only to exsist if ONTCred Cookie is missing
//Check every 15 sec for ONTCred and reload page if it
exsists. Only once though
var intRunOnce = 0;
function fctCheckONTcred () {
var allcookies = "";
allcookies = document.cookie ;
var vgnONTpos =
allcookies.inde xOf("ONTCred=") ;
if (vgnONTpos != -1){
if (intRunOnce == 0) {
topFrame.locati on.reload();
intRunOnce = 1;
}
}else{
setTimeout('fct CheckONTcred()' ,
15000);
}
}
</script>

</head>
<frameset rows="125,*" frameborder="NO " border="0" noresize
framespacing="0 ">

<frame name="topFrame" target="_top"
onload="setTime out('fctCheckON Tcred()', 15000);" scrolling="NO"
noresize src="xxxxx.html " marginheight="0 " marginwidth="0" >

<frame name="mainFrame " src="https://xxxxxxx.com/webapp/wcs/
stores/servlet/RemoteAdvancedS earchView?
langId=-1&storeId=50020 1&catalogId=500 201&manufacture r=bav99&qasFilt er=FALSE"
nor
esize marginheight="0 " marginwidth="0" >
</frameset>
</html>


<!--WEBSIDESTORY CODE HBX1.4 (SSL)-->
<!-- EM SSL - 25/Apr/2006 14:40 -->
<!--COPYRIGHT 1997-2005 WEBSIDESTORY,IN C. ALL RIGHTS RESERVED.
U.S.PATENT No.6,393,479B1 & 6,766,370. INFO:http://websidestory.com/
privacy-->
<script language="javas cript">

function hbxStrip(a)
{
a = a.split("|").jo in("");
a = a.split("&").jo in("");
a = a.split("'").jo in("");
a = a.split("#").jo in("");
a = a.split("$").jo in("");
a = a.split("%").jo in("");
a = a.split("^").jo in("");
a = a.split("*").jo in("");
a = a.split(":").jo in("");
a = a.split("!").jo in("");
a = a.split("<").jo in("");
a = a.split(">").jo in("");
a = a.split("~").jo in("");
a = a.split(";").jo in("");
a = a.split(" ").join("+" );
return a;
}

var _hbEC=0,_hbE=ne w Array;function _hbEvent(a,b){b =_hbE[_hbEC++]=new
Object();b._N=a ;b._C=0;return b;}
var hbx=_hbEvent("p v");hbx.vpc="HB X0140s";hbx.gn= "ehg-
memecinc.hitbox .com";

//BEGIN EDITABLE SECTION
//CONFIGURATION VARIABLES
hbx.acct="DM551 006HAMC;DM56022 3K8CS";//EM/GLOBAL
hbx.pn=hbxStrip ("xxxxxxxxx" );//PAGE NAME(S)
hbx.mlc="CONTEN T+CATEGORY";//MULTI-LEVEL CONTENT CATEGORY
hbx.pndef="titl e";//DEFAULT PAGE NAME
hbx.ctdef="full ";//DEFAULT CONTENT CATEGORY

//OPTIONAL PAGE VARIABLES
//ACTION SETTINGS
hbx.fv="";//FORM VALIDATION MINIMUM ELEMENTS OR SUBMIT FUNCTION NAME
hbx.lt="auto";//LINK TRACKING
hbx.dlf="n";//DOWNLOAD FILTER
hbx.dft="n";//DOWNLOAD FILE NAMING
hbx.elf="n";//EXIT LINK FILTER

//SEGMENTS AND FUNNELS
hbx.seg="";//VISITOR SEGMENTATION
hbx.fnl="";//FUNNELS

//CAMPAIGNS
hbx.cmp="";//CAMPAIGN ID
hbx.cmpn="";//CAMPAIGN ID IN QUERY
hbx.dcmp="";//DYNAMIC CAMPAIGN ID
hbx.dcmpn="";//DYNAMIC CAMPAIGN ID IN QUERY
hbx.dcmpe="";//DYNAMIC CAMPAIGN EXPIRATION
hbx.dcmpre="";//DYNAMIC CAMPAIGN RESPONSE EXPIRATION
hbx.hra="";//RESPONSE ATTRIBUTE
hbx.hqsr="";//RESPONSE ATTRIBUTE IN REFERRAL QUERY
hbx.hqsp="";//RESPONSE ATTRIBUTE IN QUERY
hbx.hlt="";//LEAD TRACKING
hbx.hla="";//LEAD ATTRIBUTE
hbx.gp="";//CAMPAIGN GOAL
hbx.gpn="";//CAMPAIGN GOAL IN QUERY
hbx.hcn="";//CONVERSION ATTRIBUTE
hbx.hcv="";//CONVERSION VALUE
hbx.cp="null";//LEGACY CAMPAIGN
hbx.cpd="";//CAMPAIGN DOMAIN

//CUSTOM VARIABLES
hbx.ci="";//CUSTOMER ID
hbx.hc1="";//CUSTOM 1
hbx.hc2="";//CUSTOM 2
hbx.hc3="";//CUSTOM 3
hbx.hc4="";//CUSTOM 4
hbx.hrf="";//CUSTOM REFERRER
hbx.pec="";//ERROR CODES

//INSERT CUSTOM EVENTS

//END EDITABLE SECTION
</script>



<script language="javas cript1.1">
//hbx.js,HBX1.5,C OPYRIGHT 1997-2005 WEBSIDESTORY,IN C. ALL RIGHTS
RESERVED. U.S.PATENT No.6,393,479B1 & 6,766,370. INFO:http://
websidestory.co m/privacy
var _vjs="HBX0150.0 1s";
var
_dl=".exe,.zip, .wav,.wmv,.mp3, .mov,.mpg,.avi, .doc,.pdf,.xls, .ppt,.gz";
function _NA(a){return new Array(a?a:0)}fu nction _NO(){return new
Object()}
var
_mn=_hbq="",_hb A=_NA(),_hud="u ndefined",_lv=_ NO(),_ec=_if=_l l=_hec=_hfs=_hf c=_fvf=_ic=_pC= _fc=_pv=0,_hbi= new
Image(),_hbin=_ NA(),_pA=_NA();
_lv.id=_lv.pos= _lv.l="";_hbE=_ D("hbE")?_hbE:" ";_hbEC=_D("hbE C")?_hbEC:
0;var _ex="expires=We d, 1 Jan 2020 00:00:00
GMT",_lvm=150,_ lidt="lid",_lpo st="lpos";
function _D(v){return(ty peof eval("window._" +v)!=_hud)?
eval("window._" +v):""}function _DD(v){return(t ypeof v!=_hud)?1:0}
function _A(v,c){return escape((_D("lc" )=="y"&&_DD(c)) ?_TL(v):v)}
function _B(){return 0}function _GP(){return
location.protoc ol=="https:"?"h ttps://":"http://"}
function _IC(a,b,c){retu rn a.charAt(b)==c? 1:0}function _II(a,b,c)
{return a.indexOf(b,c?c :0)}function _IL(a){return a!=_hud?a.lengt h:0}
function _IF(a,b,c){retu rn a.lastIndexOf(b ,c?c:_IL(a))}fu nction
_IP(a,b){return a.split(b)}
function _IS(a,b,c){retu rn b>_IL(a)?"":a.s ubstring(b,c!=n ull?
c:_IL(a))}
function _RP(a,b,c,d){d= _II(a,b);if(d>-1){a=_RP(_IS(a, 0,d)+","+_IS(a, d
+_IL(b),_IL(a)) ,b,c)}return a}
function _TL(a){return a.toLowerCase() }function _TS(a){return
a.toString()}fu nction _TV(){_hbSend() }function _SV(a,b,c)
{_hbSet(a,b,c)}
function _VS(a,b){eval(" _"+a+"='"+b+"'" )}
function _VC(a,b,c,d){b= _IP(a,",");for( c=0;c<_IL(b);c+ +)
{d=_IP(b[c],"|");_VS(d[0],(_D(d[0]))?_D(d[0]):d[1]?d[1]:"")}}
function _VL(a,b){for(a= 0;a<_hbEC;a++){ _pv=_hbE[a];if(_pv._N=="pv ")
{for(b in _pv){if(_EE(b)) {_VS(b,_pv[b])}}}}
_VC("pn|PUT+PAG E+NAME+HERE,mlc |CONTENT+CATEGO RY,elf|n,dlf|n, dft|
n,pndef|title,c tdef|full,cp|nu ll,hcn|")}_VL() ;
function _ER(a,b,c){_hbi .src=_GP()+_gn+ "/HG?hc="+_mn+"&h b="+_A(_acct)
+"&hec=1&vjs="+ _vjs+"&vpc=ERR& ec=1&err="+((ty peof a=="string")?_A (a
+"-"+c):"Unknown") }
function _EE(a){return(a !="_N"&&a!="_C" )?1:0}_EV(windo w,"error",_ER );
function _hbSend(c,a,i){ a="";_hec++;for (i in _hbA)if(typeof _hbA[i]!
="function")a+= "&"+i+"="+_ hbA[i];_Q(_hbq+"&hec= "+_hec+a
+_hbSendEV());_ hbA=_NA()}
function _hbSet(a,b,c,d, e){d=_II(_hbq," &"+a+"=");if (d>-1)
{e=_II(_hbq,"&" ,d+1);e=e>d?e:_ IL(_hbq);if(a== "n"||a=="vc on")
{_hbq=_IS(_hbq, 0,d)+"&"+a+"="+ b+
_IS(_hbq,e);_he c=-1;if(a=="n"){_p n=b}else{_mlc=b }}else{_hbq=_IS (_hbq,
0,d)+_IS(_hbq,e )}}if((a!="n")& &(a!="vcon"))_h bA[a]=(c==0)?b:_A(b) }
function _hbRedirect(a,b ,c,d,e,f,g)
{_SV("n",a);_SV ("vcon",b);if(_ DD(d)&&_IL(d)>0 ){d=_IC(d,0,"&" )?_IS(d,
1,_IL(d)):d;e=_ IP(d,"&");for(f =0;f<_IL(e);
f++){g=_IP(e[f],"=");_SV(g[0],g[1])}}_TV();if(c!= ""){_SV("he c",
0);setTimeout(" location.href=' "+c+"'",500 )}}
function _hbSendEV(a,b,c ,d,e,f,x,i){a=' ',c='',e=_IL(_h bE);for(b=0;b<e ;b
++){c=_hbE[b];for(var d in c){if(_EE(d)&&c[d].match){x=c[d].match(/\
[\]/g);
if(x!=null&&_IL (x)>c._C)c._C=_ IL(x)}}for(d in c){if(_EE(d)&&c[d].match)
{x=c[d].match(/\[\]/g);x=(x==null)? 0:_IL(x);for(i= x;i<c._C;i++)c[d]
+="[]"}}}
for(b=0;b<e;b++ ){c=_hbE[b];for(f=b+1;f<e; f++){if(_hbE[f]!
=null&&c._N==_h bE[f]._N){for(d in c){if(_EE(d)&&_ hbE[f]!=null)c[d]
+="[]"+_hbE[f][d];
_hbE[f][d]=""}}}for(d in c){if(_EE(d)&&c ._N!=""&&c._N!= "pv"){a
+="&"+c._N+"."+ d+"="+_RP(_A( c[d]),"%5B
%5D",",")}}}_hb E=_NA();_hbEC=0 ;return a}
function _hbM(a,b,c,d)
{_SV('n',a);_SV ('vcon',b);if(_ IL(c)>0)_SV(c,d );_TV()}
function _hbPageView(p,m ){_hec=-1;_hbM(p,m,"")} function _hbExitLink(n)
{_hbM(_pn,_mlc, "el",n)}functio n _hbDownload(n){ _hbM(_pn,_mlc," fn",n)}
function _hbVisitorSeg(n ,p,m){_SV("n",p );_SV("vcon",m) ;_SV("seg",n,
1);_TV()}functi on _hbCampaign(n,p ,m){_hbM(p,m,"c mp",n)}
function _hbFunnel(n,p,m ){_hbM(p,m,"fnl ",n)}functi on _hbGoalPage(n,p ,m)
{_hbM(p,m,"gp", n)}
function _hbLink(a,b,c){ _SV("lid",a);if (_DD(b))_SV("lp os",b);_TV()}
function _LE(a,b,c,d,e,f ,g,h,i,j,k,l){b ="([0-9A-Za-z\\-]*\
\.)",c=location .hostname,d=a.h ref,h='',i='';e val("__f=/"+b+"*"+b
+"/");if(_DD(__f)) {__f.exec(c);
j=(_DD(_elf))?_ elf:"";if(j!="n "){if(_II(j,"!" )>-1){h=_IS(j,
0,_II(j,"!"));i =_IS(j,_II(j,"! ")
+1,_IL(j))}else {h=j}}k=0;if(_D D(_elf)&&_elf!= "n"){
if(_IL(i)){l=_I P(i,",");for(g= 0;g<_IL(l);g+
+)if(_II(d,l[g])>-1)return}if(_IL (h)){l=_IP(h,", ");for(g=0;g<_I L(h);g+
+)if(_II(d,l[g])>-1)k=1}}
if(_II(a.hostna me,RegExp.$2)<0 ||k){ e=_IL(d)-1;return _IC(d,e,'/')?
_IS(d,0,e):d}}}
function _LD(a,b,c,d,e,f ){b=a.pathname, d='',e='';b=_IS (b,_IF(b,"/")
+1,_IL(b));c=(_ DD(_dlf))?_dlf: "";if(c!="n"){i f(_II(c,"!")>-1){d=","+
_IS(c,0,_II(c," !"));e=","+_IS( c,_II(c,"!")
+1,_IL(c))}else {d=","+c}}f=_II (b,"?");b=(f>-1)?_IS(b,
0,f):b;if(_IF(b ,".")>-1){f=_IS(b,_IF( b,"."),_IL(b) );
if(_II(_dl+d,f) >-1&&_II(e,f)<0){ var dl=b;if(_DD(_df t))
{if(_dft=="y"&& a.name){dl=a.na me}else if(_dft=="full" )
{dl=a.pathname} }return dl}}}
function _LP(a,b,c){for( c=0;c<_IL(a);c+ +){if(b==0)
{if(_IL(_lv.l)< _lvm)_LV(a[c]);else break}else
if(b==1)_EV(a[c],'mousedown',_L T)}}
function _LV(a,b,c){b=_L N(a);c=b[0]+b[1];if(_IL(c)){_lv .id+=_A(b[0])
+",";_lv.pos+=_ A(b[1])+",";_lv.l+=c} }
function _LN(a,b,c,d){b= a.href;b+=a.nam e?
a.name:"";c=_LV P(b,_lidt);d=_L VP(b,_lpost);re turn[c,d]}
function _LT(e){if((e.wh ich&&e.which==1 )||(e.button&&e .button==1)){va r
a=document.all? window.event.sr cElement:this;f or(var i=0;i<4;i++)
{if(a.tagName&&
_TL(a.tagName)! ="a"&&_TL(a.tag Name)!="area"){ a=a.parentEleme nt}}var
b=_LN(a),c='',d ='';a.lid=b[0];a.lpos=b[1];if(_D("lt")&&_ lt!="manual")
{if((a.tagName& &
_TL(a.tagName)= ="area")){if(!_ IL(a.lid)){if(a .parentNode)
{if(a.parentNod e.name)a.lid=a. parentNode.name ;else
a.lid=a.parentN ode.id}}if(!_IL (a.lpos))
a.lpos=a.coords }else{if(_IL(a. lid)<1)a.lid=_L S(a.text?
a.text:a.innerT ext?a.innerText :"");if(!_IL(a. lid)||
_II(_TL(a.lid), "<img")>-1)a.lid=_LI(a)} }
if(!_IL(a.lpos) &&_D("lt")=="au to_pos"&&a.tagN ame&&_TL(a.tagN ame)!
="area"){c=docu ment.links;for( d=0;d<_IL(c);d+ +){if(a==c[d]){a.lpos=d
+1;break}}}
var _f=0,j='',k='', l=(a.protocol)? _TL(a.protocol) :"";
if(l&&l!="mailt o:"&&l!="javasc ript:")
{j=_LE(a),k=_LD (a);if(_DD(k))a .fn=k;else if(_DD(j))a.el= j}
if(_D("lt")&&_I C(_lt,0,"n")!=1 &&_DD(a.lid)&&_ IL(a.lid)>0)
{_SV("lid",a.li d);if(_DD(a.lpo s))_SV("lpos",a .lpos);_f=1}if( _DD(a.fn))
{_SV("fn",a.fn) ;_f=2}
else if(_DD(a.el)){_ SV("el",a.el);_ f=1}if(_f>0){_T V()}}}
function _LVP(a,b,c,d,e) {c=_II(a,"&"+b+ "=");c=c<0?_II( a,"?"+b
+"="):c;if(c >-1){d=_II(a,'&', c+_IL(b)+2);e=_ IS(a,c+_IL(b)+2 ,d>-1?
d:_IL(a));
if(!_ec){if(!(_ II(e,"//")==0))retu rn e}else return e}return ""}
function _LI(a){var
b=""+a.innerHTM L,bu=_TL(b),i=_ II(bu,"<img");i f(bu&&i>-1){eval("__f=/
src\s*=\s*['\"]?([^'\" ]+)['\"]?/i");__f.exec(b) ;
if(RegExp.$1)b= RegExp.$1}retur n b}
function _LSP(a,b,c,d){d =_IP(a,b);retur n d.join(c)}
function _LS(a,b,c,d,e,f ,g){c=_D("lim") ?_lim:100;b=(_I L(a)>c)?_A(_IS( a,
0,c)):_A(a);b=_ LSP(b,"%0A","%2 0");b=_LSP(b,"% 0D","%20");b=_L SP(b,"%09","%20 ");
c=_IP(b,"%20"); d=_NA();e=0;for (f=0;f<_IL(c);f ++)
{g=_RP(c[f],"%20","");if(_ IL(g)>0){d[e++]=g}}b=d.join("% 20");return
unescape(b)}
function _EM(a,b,c,d)
{a=_D("fv");b=_ II(a,";"),c=par seInt(a);d=3;if (_TL(a)=="n")
{d=999;_fv=""}e lse if(b>-1){d=_IS(a,0,b) ;_fv=_IS(a,b+1, _IL(a))}
else if(c>0){d=c;_fv =""}return d}
function _FF(e){var a=(_bnN)?this:_ EVO(e);_hlf=(a. lf)?a.lf:""}
function _FU(e){if(_hfs= =0&&_IL(_hlf)>0 &&_fa==1){_hfs= 1;if(_hfc)
{_SV("sf","1")} else if(_IL(_hlf)>0)
{_SV("lf",_hlf) }_TV();_hlf="", _hfs=0,_hfc=0}}
function _FO(e){var
a=true;if(_DD(t his._FS))eval(" try{a=this._FS( )}catch(e){}"); if(a!
=false)_hfc=1;r eturn a}
function _FA(a,b,c,d,e,f ,g,h,i,ff,fv,s) {b=a.forms;ff=n ew
Object();f=_EM( );for(c=0;c<_IL (b);c++)
{ff=b[c],d=0,s=0,e=ff.e lements,fv=eval (_D("fv"));
if(_DD(fv)&&_TL (_TS(fv))!="n"& &fv!=""&&typ eof fv=="function") {_fv=new
Function("if("+ _fv+"())
{_fvf=0;_hfc=1} ");_EV(ff,"subm it",_fv),_fvf=1 ,_fa=1}
g=ff.name?ff.na me:"forms["+c+"]";for(h=0;h<_IL (e);h++)
{if(e[h].type&&"hiddenb uttonsubmitimag ereset".indexOf (e[h].type)<0&&d+
+>=f)break}if(d >=f){_fa=1;
for(h=0;h<_IL(e );h++)
{i=e[h];if(i.type&&"hi ddenbuttonsubmi timagereset".in dexOf(i.type)<0 )
{i.lf=g+".";i.l f+=(i.name&&i.n ame!="")?i.name :"elements["+h+"]";
_EV(i,"focus",_ FF)}}ff._FS=nul l;ff._FS=ff.ons ubmit;if(_DD(ff ._FS)&&ff._FS!
=null){ff.onsub mit=_FO}else if(!(_bnN&&_bv< 5)&&_hM&&!(_bnI &&!_I5))
{if((!_bnI)||
(_II(navigator. userAgent,"Oper a")>-1))
{ff.onsubmit=_F O}else{_EV(ff," submit",_FO);
eval("try{docum ent.forms["+c+"]._FS=document.f orms["+c
+"].submit;documen t.forms["+c+"].submit=_FO;thr ow ''}catch(E){}") }}}}}
function _GR(a,b,c,d){if (!_D("hrf"))ret urn a;if(_II(_hrf," http",
0)>-1)return _hrf;b=window.l ocation.search; b=_IL(b)>1?_IS( b,
1,_IL(b)):"";
c=_II(b,_hrf+"= ");if(c>-1){ d=_II(b,"&",c+1 );d=d>c?d:_IL(b );b=_IS(b,c
+_IL(_hrf)+1,d) }return(b!=_hud &&_IL(b)>0)?b:a }
function _PO(a,b,c,d,e,f ,g){d=location, e=d.pathname,f= _IS(e,_IF(e,"/")
+1),g=document. title;if(a&&b== c){return(_pnde f=="title"&&g!= ""&&g!=d&&
!(_bnN&&_II(g," http")>0))?g:f? f:_pndef}else{r eturn b==c?(e==""||
e=="/")?"/":_IS(e,(_ctdef !="full")?
_IF(e,"/",_IF(e,"/")-2):_II(e,"/"),_IF(e,"/"))
:(b=="/")?b:((_II( b,"/")?"/":"")+(_IF( b,"/")==_IL(b)-1?_IS(b,
0,_IL(b)-1):b))}}
function _PP(a,b,c,d){re turn ""+(c>-1?_PO(b,_IS(a,0 ,c),d)
+";"+_PP(_IS(a, c+1),b,_II(_IS( a,c+1),";")):_P O(b,a,d))}
_mlc=_PP(_mlc,0 ,_II( _mlc,";"),"CONT ENT+CATEGORY"); _pn=_PP(_pn,
1,_II(_pn,";"), "PUT+PAGE+NAME+ HERE");
function _NN(a){return _D(a)!="none"}i f(_NN("lt")){_L P(document.link s,
0)}
function _E(a){var b="";var d=_IP(a,",");fo r(var c=0;c<_IL(d);c+ +)b
+="&"+d[c]+"="+_A(_D(d[c]));return b}
function _F(a,b){return( !_II(a,"?"+b+"= "))?0:_II(a,"&" +b+"=")}functio n
_G(a,b,c,d){var e=_F(a,b);if(d& &e<0&&top&&wind ow!=top){e=_F(_ tls,b);
if(e>-1)a=_tls};retur n(e>-1)?_IS(a,e+2+_I L(b),(_II(a,"&" ,e+1)>-1)?
_II(a,"&",e+1): _IL(a)):c}
function _H(a,b,c){if(!a )a=c;if(_I5||_N 6)
{eval("try{_vv= _G(location.sea rch,'"+a+"','"+ b+"',1)}"+__c
+"{}")}else{_vv =_G(location.se arch,a,b,1)}ret urn unescape(_vv)}
function _I(a,b,c,d){__f =_IS(a,_II(a,"? "));if(b){if(_I 5||_N6)
{eval("try{_hra =_G(__f,_hqsr,_ hra,0)}"+__c
+"{}")}else{_hr a=_G(__f,_hqsr, _hra,0)}};
if(c&&!_hra){if (_I5||_N6){eval ("try{_hra=_G(l ocation.search, _hqsp,_hra,
1)}"+__c+"{}")} else{_hra=_G(lo cation.search,_ hqsp,_hra,1)}};
if(d&&!_hra)_hr a=d;return _hra}function _J(a,b,c,d)
{c=_II(a,"CP=") ;d=_II(a,b,c+3) ;return(c<0)?"n ull":_IS(a,c+3, (d<0)?
_IL(a):d)}
var
__r=".referrer" ,_rf=_A(eval("d ocument"+__r)), _et=0,_oe=0,_we =0,_ar="",_hM=( !
(_II(navigator. userAgent,"Mac" )>-1)),_tls="";
_bv=parseInt(na vigator.appVers ion);_bv=(_bv>9 9)?(_bv/100):_bv;var
__f,_hrat=_D("h ra"),_hra="",__ c="catch(_e)",_ hbi=new
Image(),_fa=0,_ hlfs=0,_hoc=0,
_hlf='',_ce='', _ln='',_pl='',_ bn=navigator.ap pName,_bn=(_II( _bn,"Microsoft" )?
_bn:"MSIE"),_bn N=(_bn=="Netsca pe"),_bnI=(_bn= ="MSIE"),
_hck="*; path=/; "+(_D("cpd")&&_ D("cpd")!=""? (" domain=."+_D("c pd")+";
"):"")
+_ex,_N6=(_bnN& &_bv>4),_I5=fal se,_ss="na",_sc ="na",_sv=11,_c y="u",_hp="u" ,
_tp=_D("ptc");i f(_bn=="MSIE"){ var
_nua=navigator. userAgent,_is=_ II(_nua,_bn),_i f=_II(_nua,".", _is);if(_if>_is )_I5=_nua.subst ring(_is
+5,_if)>=5}
if(_N6||_I5)eva l("try{_tls=top .location.searc h}catch(_e){}")
function _PV()
{_dcmpe=_H(_D(" dcmpe"),_D("dcm pe"),"DCMPE");_ dcmpre=_H(_D("d cmpre"),_D("dcm pre"),"DCMPRE") ;_vv="";_cmp=_H (_D("cmpn"),_D( "cmp"),"CMP ");
_gp=_H(_D("gpn" ),_D("gp"),"GP" );_dcmp=_H(_D(" dcmpn"),_D("dcm p"),"DCMP");if( _II(_cmp,"SFS-")>-1)
{document.cooki e="HBCMP="+_cmp +"; path=/;"+
(_D("cpd")&&_D( "cpd")!=""? (" domain=."+_D("c pd")+"; "):"")
+_ex}if(_bnI&&_ bv>3)_ln=naviga tor.userLanguag e;
if(_bnN){if(_bv >3)_ln=navigato r.language;if(_ bv>2)for(var
i=0;i<_IL(navig ator.plugins);i ++)_pl+=navigat or.plugins[i].name
+":"};_cp=_D("c p");
if(location.sea rch&&_TL(_cp)== "null")_cp=_J(l ocation.search, "&");if(_II(doc ument.cookie,"C P=")>-1)
{
_ce="y";_hd=_J( document.cookie ,"*");if(_TL(_h d)!="null"&&_cp =="null")
{_cp=_hd}else{d ocument.cookie= "CP="+_cp
+_hck}}else{doc ument.cookie="C P="+_cp+_hck;
_ce=(_II(docume nt.cookie,"CP=" )>-1)?"y":"n"};if( window.screen)
{_sv=12;_ss=scr een.width+"*"+s creen.height;_s c=_bnI?
screen.colorDep th:screen.pixel Depth;
if(_sc==_hud)_s c="na"};_ra=_NA ();if(_ra.toSou rce||
(_bnI&&_ra.shif t))_sv=13;if(_I 5&&_hM)
{if(_II(""+navi gator.appMinorV ersion,"Privacy ")>-1)_ce="p";
if(document.bod y&&document.bod y.addBehavior)
{document.body. addBehavior("#d efault#homePage ");_hp=document .body.isHomePag e(location.href )?"y":"n";
document.body.a ddBehavior("#de fault#clientCap s");_cy=documen t.body.connecti onType}};var
_hcc=(_DD(_hcn) )?_D("hcv"):""; if(!_D("gn"))_g n="ehg.hitbox.c om";
if(_D("ct")&&!_ D("mlc"))_mlc=_ ct;_ar=_GP()+_g n+"/HG?hc="+_mn
+"&hb="+_A(_acc t)+"&cd=1&hv=6& n="+_A(_pn,1)+" &con=&vcon="+_A (_mlc,
1)+"&tt="+_D("l t")+
"&ja="+(navigat or.javaEnabled( )?"y":"n")+"&dt ="+(new Date()).getHour s()
+"&zo="+(new Date()).getTime zoneOffset()
+"&lm="+Date.pa rse(document.la stModified)
+(_tp?("&pt="+_ tp):"")+_E((_bn N?"bn,":"")
+"ce,ss,sc,sv,c y,hp,ln,vpc,vjs ,hec,pec,cmp,gp ,dcmp,dcmpe,dcm pre,cp,fnl")
+"&seg="+_D("se g")+"&epg="+_D( "epg")+
"&cv="+_A(_hcc) +"&gn="+_A(_D(" hcn"))+"&ld="+_ A(_D("hlt"))
+"&la="+_A(_D(" hla"))+"&c1="+_ A(_D("hc1"))+"& c2="+_A(_D("hc2 "))
+"&c3="+_A(_D(" hc3"))+"&c4="+
_A(_D("hc4"))+" &customerid="+_ A(_D("ci")?_ci: _D("cid"))
+"&lv.id="+_lv. id+"&lv.pos="+_ lv.pos+"&ttt="+ _lidt+","+_lpos t;
if(_I5||_N6){ev al("try{_rf=_A( top.document"+_ _r+")+''}"+__c
+"{_rf=_A(docum ent"+__r+")+''} ")}
else{if(top.doc ument&&_IL(pare nt.frames)>1)
{_rf=_A(eval("d ocument"+__r))+ ""}else if(top.document )
{_rf=_A(eval("t op.document"+__ r))+""}}if((_rf ==_hud)||
(_rf==""))_rf=" bookmark";_rf=u nescape(_rf);_r f=_GR(_rf);_hra =_I(_rf,_D("hqs r"),_D("hqsp"), _hrat);_ar
+="&ra="+_A(_hr a)+"&rf="+_A(_I S(_rf,0,500))+
"&pl="+_A(_pl)+ _hbSendEV();if( _D("onlyMedia") !="y")_hbi.src= _ar
+"&hid="+Math.r andom();_hbq=_I S(_ar,
0,_II(_ar,"&hec "));_hbE=_NA()} _PV();
function _Q(a){var b="";b=new Image();b.src=a +"&hid="+Math.r andom()}
function __X(a){if(_ec== 0){_ec=1;a=docu ment;if(_NN("lt ")||_NN("dlf")| |
_NN("elf")){_LP (a.links,1)}if( _NN("fv"))_FA(a )}}
function _EV(a,b,c){if(a .addEventListen er)
{a.addEventList ener(b,c,false) }else if(a.attachEven t)
{a.attachEvent( "on"+b,c)}}
function _EVO(e){return document.all?wi ndow.event.srcE lement:this}
_EV(window,"loa d",__X);_EV(win dow,"unload",_F U);eval('setTim eout("__X()",
3000)');
</script>
<!--END WEBSIDESTORY CODE-->

[Jerry Stuckle]

Re: parsing website with a &quot;searching ....&quot; page

Aaron wrote:

On Apr 6, 10:55 am, Erwin Moller
<since_humans_r ead_this_I_am_s pammed_too_m... @spamyourself.c omwrote:

>Aaron wrote:

>>I'm trying to parse a table on a webpage to pull down some data I
>>need. The page is based off of information entered into a form. when
>>you submit the data from the form it displays a "Searching. .." page
>>then, refreshes and displays the table I want. I have code that grabs
>>data from the page using cURL but when I look at the data it contains
>>the "Searching. .." page and not the table that I want. below is the
>>code i have so far....Thanks in advance for any help.
>><?php
>> $url="http://www.website.com ";
>> $post_data = array();
>> $post_data['postvar1'] = "val1";
>> $post_data['postvar2'] = "val2";
>> $o="";
>> foreach($post_d ata as $k=>$v)
>> {
>> $o.= "$k=".utf8_enco de($v)."&";
>> }
>> $post_data=subs tr($o,0,-1);
>> $ch= curl_init();
>> curl_setopt($ch , CURLOPT_POST,1) ;
>> curl_setopt($ch , CURLOPT_HEADER, 0);
>> curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
>> curl_setopt($ch , CURLOPT_URL,$ur l);
>> curl_setopt($ch , CURLOPT_POSTFIE LDS, $post_data);
>> $result = curl_exec($ch);
>> curl_close($ch) ;
>> $result=explode ("\n",$resul t);
>>?>

>Hi,
>>
>The page is probably using JavaScript to give that effect.
>Inspect the content of $result to see if this is the case.
>Look for divs that are not visible and made visible when the page loads (end
>of script, or an onLoad event).
>The data you want might very well be inside the $result.
>>
>If not, give more information WHAT the $result contained.
>>
>Regards,
>Erwin Moller- Hide quoted text -
>>
>- Show quoted text -

>
Heres basicly what was returned by $result
>

<snipped lots of code>

Erwin is correct. That's using a LOT of javascript, plus it's using
frames. This one is going to be very tough to scrape - you'll need to
decode what the javascript does and emulate it with Curl to get the page.

But it may also be that the webmaster implemented this in part to keep
anyone from scraping the screen. Most sites do not like this.

You'd be better off contacting the owner and seeing if there is another
way to get the information.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===