STRANGE QUESTION : search engine query and web site dowload

[igthibau]

Hello everyone,
well, the title says pretty much all.
First off though I'd like to say I have no idea what ng I should post
this to. I chose this one because I am more likely to understand
c-based answers than other language's.
So, my problem is simple : to submit a query to a search engine, one goes to
, say www.google.com, enters keywords and a long list appears. Then one
clicks onto the appropriate hyperlink to go to the chosen site.
Question : how does one do that using scripts / fortran / C / html /
whatnot. As in : using command line interfacing to as to enable, say, loops
or non-graphical interactivity?
The idea is to be able to, within a program, call up a search engine, run it
using keywords and then download the first X sites.

There, I said it. I have no idea where to start from let alone what langage
to use and would hugely appreciate any help whatsoever !!!

thanks to all.
G.

[Moonlit]

Re: STRANGE QUESTION : search engine query and web site dowload

Hi,


"igthibau" <igthibau@wanad oo.fr> wrote in message
news:bnfug9$ihe $1@news-reader3.wanadoo .fr...[color=blue]
> Hello everyone,
> well, the title says pretty much all.
> First off though I'd like to say I have no idea what ng I should post
> this to. I chose this one because I am more likely to understand
> c-based answers than other language's.
> So, my problem is simple : to submit a query to a search engine, one goes[/color]
to[color=blue]
> , say www.google.com, enters keywords and a long list appears. Then one
> clicks onto the appropriate hyperlink to go to the chosen site.
> Question : how does one do that using scripts / fortran / C / html /
> whatnot. As in : using command line interfacing to as to enable, say,[/color]
loops[color=blue]
> or non-graphical interactivity?
> The idea is to be able to, within a program, call up a search engine, run[/color]
it[color=blue]
> using keywords and then download the first X sites.[/color]

Well would you should do is create a connection to the server (port 80)
using a socket. Then you would send information to it and read from it. The
protocol used is described in the http rfc (google for it). You could then
create a query and scan the pages for the results. Of course you have to
look at the pages to see how the results are returned.

There is also a library available somewhere that let you build for instance
a browser, It has al kinds of http function which might save you a lot of
time.

I actually wrote an app that talks to our local SMS http server in the
company to send sms to mobile phones. I just intercepted all trafic from my
webbrowser to the site and back to see how they communicated, then I wrote a
class that implemented a very small subset of the http protocol.. Such an
approach might work for you too.

[color=blue]
>
> There, I said it. I have no idea where to start from let alone what[/color]
langage[color=blue]
> to use and would hugely appreciate any help whatsoever !!!
>
> thanks to all.
> G.[/color]

Regards, Ron AF Greve.

[Jerry Coffin]

Re: STRANGE QUESTION : search engine query and web site dowload

In article <bnfug9$ihe$1@n ews-reader3.wanadoo .fr>, igthibau@wanado o.fr
says...

[ ... ]
[color=blue]
> The idea is to be able to, within a program, call up a search engine, run it
> using keywords and then download the first X sites.[/color]

There's probably some newsgroup related to network development on your
chosen platform where this is more topical, but if you go to:

http://www.google.com/apis/

you can probably bypass all that.


--
Later,
Jerry.

The universe is a figment of its own imagination.

[igthibau]

Re: STRANGE QUESTION : search engine query and web site dowload

Hmmmmm not sure I understood all that, but it looks as though I am embarking
on network programming...
on which I have not a clue.
any book / reference I might find useful ?

thanks
G.

Moonlit wrote:
[color=blue]
> Hi,
>
>
> "igthibau" <igthibau@wanad oo.fr> wrote in message
> news:bnfug9$ihe $1@news-reader3.wanadoo .fr...[color=green]
>> Hello everyone,
>> well, the title says pretty much all.
>> First off though I'd like to say I have no idea what ng I should post
>> this to. I chose this one because I am more likely to understand
>> c-based answers than other language's.
>> So, my problem is simple : to submit a query to a search engine, one goes[/color]
> to[color=green]
>> , say www.google.com, enters keywords and a long list appears. Then one
>> clicks onto the appropriate hyperlink to go to the chosen site.
>> Question : how does one do that using scripts / fortran / C / html /
>> whatnot. As in : using command line interfacing to as to enable, say,[/color]
> loops[color=green]
>> or non-graphical interactivity?
>> The idea is to be able to, within a program, call up a search engine, run[/color]
> it[color=green]
>> using keywords and then download the first X sites.[/color]
>
> Well would you should do is create a connection to the server (port 80)
> using a socket. Then you would send information to it and read from it.
> The protocol used is described in the http rfc (google for it). You could
> then create a query and scan the pages for the results. Of course you have
> to look at the pages to see how the results are returned.
>
> There is also a library available somewhere that let you build for
> instance a browser, It has al kinds of http function which might save you
> a lot of time.
>
> I actually wrote an app that talks to our local SMS http server in the
> company to send sms to mobile phones. I just intercepted all trafic from
> my webbrowser to the site and back to see how they communicated, then I
> wrote a class that implemented a very small subset of the http protocol..
> Such an approach might work for you too.
>
>[color=green]
>>
>> There, I said it. I have no idea where to start from let alone what[/color]
> langage[color=green]
>> to use and would hugely appreciate any help whatsoever !!!
>>
>> thanks to all.
>> G.[/color]
>
> Regards, Ron AF Greve.[/color]

[Moonlit]

Re: STRANGE QUESTION : search engine query and web site dowload

Hi,

"igthibau" <igthibau@wanad oo.fr> wrote in message
news:bnheu6$1tg $1@news-reader2.wanadoo .fr...[color=blue]
> Hmmmmm not sure I understood all that, but it looks as though I am[/color]
embarking[color=blue]
> on network programming...[/color]
You sure do ;-)

About sockets

http://www.developerweb.net/sock-faq/flatfaq.php

Don't know where but if you google for RFC and HTTP you probably find lots
of info on the hypertext protocol.

The first thing you might try if you have written the basics (a simple
network connection) is sending the following 4 lines (2 empty returns at the
end) Host should be the host you connect. Use port 80. (You could actually
try this with telnet <host> 80). This should get you back the main page of
that site.

GET / HTTP1/1
host: <fill in the server to connect to>



Regards, Ron AF Greve.



[color=blue]
> on which I have not a clue.
> any book / reference I might find useful ?
>
> thanks
> G.
>
> Moonlit wrote:
>[color=green]
> > Hi,
> >
> >
> > "igthibau" <igthibau@wanad oo.fr> wrote in message
> > news:bnfug9$ihe $1@news-reader3.wanadoo .fr...[color=darkred]
> >> Hello everyone,
> >> well, the title says pretty much all.
> >> First off though I'd like to say I have no idea what ng I should post
> >> this to. I chose this one because I am more likely to understand
> >> c-based answers than other language's.
> >> So, my problem is simple : to submit a query to a search engine, one[/color][/color][/color]
goes[color=blue][color=green]
> > to[color=darkred]
> >> , say www.google.com, enters keywords and a long list appears. Then one
> >> clicks onto the appropriate hyperlink to go to the chosen site.
> >> Question : how does one do that using scripts / fortran / C / html /
> >> whatnot. As in : using command line interfacing to as to enable, say,[/color]
> > loops[color=darkred]
> >> or non-graphical interactivity?
> >> The idea is to be able to, within a program, call up a search engine,[/color][/color][/color]
run[color=blue][color=green]
> > it[color=darkred]
> >> using keywords and then download the first X sites.[/color]
> >
> > Well would you should do is create a connection to the server (port 80)
> > using a socket. Then you would send information to it and read from it.
> > The protocol used is described in the http rfc (google for it). You[/color][/color]
could[color=blue][color=green]
> > then create a query and scan the pages for the results. Of course you[/color][/color]
have[color=blue][color=green]
> > to look at the pages to see how the results are returned.
> >
> > There is also a library available somewhere that let you build for
> > instance a browser, It has al kinds of http function which might save[/color][/color]
you[color=blue][color=green]
> > a lot of time.
> >
> > I actually wrote an app that talks to our local SMS http server in the
> > company to send sms to mobile phones. I just intercepted all trafic from
> > my webbrowser to the site and back to see how they communicated, then I
> > wrote a class that implemented a very small subset of the http[/color][/color]
protocol..[color=blue][color=green]
> > Such an approach might work for you too.
> >
> >[color=darkred]
> >>
> >> There, I said it. I have no idea where to start from let alone what[/color]
> > langage[color=darkred]
> >> to use and would hugely appreciate any help whatsoever !!!
> >>
> >> thanks to all.
> >> G.[/color]
> >
> > Regards, Ron AF Greve.[/color]
>[/color]