code to fetch data from online source

[cobra2341]

Sorry.
Ok, I read on LWP and it doesn't seem to be what I want. What I am wanting to do is have perl download the text from the site so I can import it to a database.
Thanks

[numberwhun]

Well, any way you look at it, what you are going to be downloading is the web page itself, which includes all of the HTML. Now, you could use LWP::Simple 's get function to fetch the web page, but you are going to have to parse out the text with a regex, or go so far as to use a module that helps to parse HTML into its elements (I would look one up, but am exhausted and on the way to bed at the moment).

Regards,

Jeff

[KevinADC]

Yeah some people aren't very tolerant here. It seems that they are only interested in answering questions that "they" find interesting. Not much concern for the person who's posting's needs.

And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.

Personally, I have no concern at all for the person posting the question. I answer questions because I want to, for my own reasons, not for anyone elses reasons or for anyone elses "needs". It's all for my own selfish benefit. If it were not for my benefit, I would never answer one single question on this or any other forum. And because it is for my own selfish benefit, I am very prolific in this regards. I have helped literally thousands of people over the years on forums in a variety of disciplines, but these days it's 99% perl that interests me. If you do something like this for the other persons benefit or based on their needs, you will help a few people at best.

If it were not for my own benefit why would I continue? I'm not here to make some sort of sacrifice of myself and become the willing servant of other peoples needs. THEY COME HERE TO FULFILL MY NEEDS (upper-case for emphasis only). A concept you may not understand. I get the most benefit out of the whole deal, and they get some benefit as a sort of accidental side affect, another concept you may find hard to wrap your head around.

I give all questions the exact same priority and attention: none. If I want to answer a question, I will. If it iterests me, all the better chance I will answer it. If it is asked intelligently, all the better chance of being answered. If it shows real effort by the person asking, all the better it will be answered.

But as I said, I do it all for my own benefit, for my own selfish reasons, nobody elses reasons and nobody elses benefit. Reasons that are none of yours or anyone elses business unless I choose to share my reasons with them.

If you or anyone else answers questions on forums because of the benefit the other person receives, or based on the other persons needs, I hope you get a warm and fuzzy feeling all over. My opinion is that anyone that answers questions for those reasons is a clueless i-d-i-o-t or just too young to know better.

That same philosophy applies for nearly everything else in life. First and foremost: do something that will benefit you (as long as it does not cause harm to others that you can reasonably be expected to know about) and everyone else will benefit, either directly or indirectly.

Regards,
Kevin

[KevinADC]

Sorry.
Ok, I read on LWP and it doesn't seem to be what I want. What I am wanting to do is have perl download the text from the site so I can import it to a database.
Thanks

See if the site has an RSS feed.

[cobra2341]

No the site doesn't have an rss feed.

[numberwhun]

And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.

I have to, in addition to my earlier statement, second Kevin.

Since dedecating 99% of my time to Perl, I have spent a lot of time in this forum answering questions. More than once, I have found myself researching a module on cpan just to figure out why someone's code wasn't working.

Sure, I got the script working as expected, but the biggest benefit was actually to me. I learned a new Perl module and a few other tricks at the same time. Alsdo, more than once has either Kevin or Miller threw in their .02 and I learned a whole new way to do something.

In your tunnel-visioned view, you see us as rude and not caring when we are actually asking for the person's code for a reason; they want to learn and usd writing your script or doing your homework will not benefit you at all.

Besides, you thing we are rude? Try going over to perlmonks.org and posting questions without the necessasry pertinent information. You will either be sent a link to the posting guidelines, or told to revamp your question to be more informative and measningful.

Regards,

Jeff

[KevinADC]

No the site doesn't have an rss feed.

Rats! That would have been too easy. ;)

[cobra2341]

ok, so what do I need to learn in order to do what I want?
Thanks

[docdiesel]

Hi cobra,

the question "what do I need to learn" forces another question: what do you know so far? Do you already have experience in programming, maybe in another language like C or PHP? Then you'll need to learn Perl, otherwise you'll need to learn programming.

I assume that the latter one is the case. Let's have a look at some simple code:

#!/usr/bin/perl
# top line should be the path to your perl interpreter
# on Unix systems; I think you won't need it on windows

# 1) define basic settings and modules to be used 1st.
use strict;
use LWP::UserAgent;

### 2) then set your definitions/constants

# use "[I]my varname[/I]" for first definition
#-- URL to the web page
my $myurl = 'http://www.xxxsite.com/change/this';

#-- 3) define misc. variables
my $myagent;
my $myrequest;
my $myresult;


### 4) then comes the code

#-- create an http user agent
$myagent = new LWP::UserAgent;


#-- Prepare & send request
$myrequest = new HTTP::Request GET => $myurl;
$myresult  = $myagent->request($myrequest);

#-- check if everything went fine
#-- if yes, write to stdout
#-- if no, die & write error msg
if( $myresult->is_success )
{
  print $myrequest->content;
} 
else
{
  die "HTTP request failed.";
}

Btw., do you know about Perl modules and how to handle them? Anyway, I agree to that point that you should look for some tutorials like at www.tizag.com or similar.

Regards,

Bernd

[cobra2341]

I am doing MySQL right now, and was going to use this data for a database. I really didn't want to learn alot about perl, I just wanted to use it to get this data. I have no idea what that code did. I need the data to actually download so I can use it for a database, not just say success or die. Thanks

[numberwhun]

While I do understand your situation, this however, is not a code writing service, it is a learning forum. Granted, there are people here who will happily take your $$$ and write you the script that you are looking for, but I doubt you will find anyone to just write it or finish it for you.

Regards,

Jeff

[cobra2341]

I understand. I am going to finsh learning MySQL then I'll try some Perl. Thanks for all the help

[KevinADC]

I understand you are not wanting to learn perl, but if you had taken maybe 15 minutes to read the LWP::UserAgent documentation I'm sure you could have figured it out.

[docdiesel]

Hi,

I am doing MySQL right now, and was going to use this data for a database. I really didn't want to learn alot about perl, I just wanted to use it to get this data. I have no idea what that code did. I need the data to actually download so I can use it for a database, not just say success or die. Thanks

well, it was a simple example which you could have based a script with your needs on. Nevermind, can't always be a success.

If that data to be downloaded is right in the format to import into mysql by using its command line interface, you may try wget also (see http://en.wikipedia.org/wiki/Wget ).

Write a batch to wget' the data and run mysql cmd line on the downloaded file subsequently. For that mysql is something you know I guess you won't need help there.

Otherwise, have fun learning perl.

Regards,

Bernd

[cobra2341]

I did read the LWP and I didn't think it was what I needed. Thanks