parsing internet page using C

**RedSon** · Mar 28 '07, 05:45 PM

Try searching for web robots or web spiders.

**Jan Spatina** · Mar 28 '07, 07:54 PM

Originally posted by manontheedge

I've been trying to parse data on a web page in C, but after a several hours of searching the internet for help with writing this code I'm still lost.

Can anyone give me any tips, code, or any other sort of help with this? Nothing I try seems to work and I'm just extremely frustrated and would appreciate any help.

What I'm trying to do is go to a web site, look for various keywords, and based on what it finds, pull that data out and either put it in variables in the code or put it in a text document.

Hi,
you need to establish http connection using sockets, then send something like "GET /web/index.html HTTP/1.0\nhost: www.sapik.cz\n\ n" and get the html code in reply and parse. This won't work for php sites. Maybe I can write some example if this is what you need.

-jan-

**manontheedge** · Mar 28 '07, 08:45 PM

that is what I need...and they are html pages. If you can post some code that would be a great help...I've gotten pretty much nowhere with this. I was about to settle for opening the page, copying it to excel and sorting through it there due to my frustration, so yes I would appreciate the help very much.

**Jan Spatina** · Mar 29 '07, 05:38 PM

Originally posted by manontheedge

that is what I need...and they are html pages. If you can post some code that would be a great help...I've gotten pretty much nowhere with this. I was about to settle for opening the page, copying it to excel and sorting through it there due to my frustration, so yes I would appreciate the help very much.

Hi, this is the code:

Code:

 #include <iostream>
#include <string>
#include <algorithm>
#include <fstream>
#include <windows.h>

using namespace std;

int t_sockets::GetWeb()
{
    #define BUFSIZE 1000000
    WORD wVersionRequested = MAKEWORD(1,1); 
    WSADATA data;                           // library
    string text("GET /forum/thread624279.html HTTP/1.0\nhost: www.thescripts.com\n\n");
    hostent *host;                          // remote machine
    sockaddr_in serverSock;                 // remote socket
    int mySocket;                           // Socket    
    char buf[BUFSIZE];                      // input buffer
    int size, totalSize = 0;                // number of recieved and sent bytes
    ofstream output(".\\download.html");    //write html data 
    // get sockets ready
    if (WSAStartup(wVersionRequested, &data) != 0)
    {
        cerr << "error in inicialization of sockets" << endl;
        
        return -1;
    }    
    // get info about remote machine
    if ((host = gethostbyname("www.builder.cz")) == NULL)
    {
        cerr << "Wrong address" << endl;
        WSACleanup();
        return -1;
    }
    // Creation of a socket
    if ((mySocket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == -1)
    {
        cerr << "Can't create socket" << endl;
        WSACleanup();
        return -1;
    }
    // fill in sockaddr_in
    // 1) protocol family
    serverSock.sin_family = AF_INET;
    // 2) port to connect (http)
    serverSock.sin_port = htons(80);
    // 3) IP address to connect to
    memcpy(&(serverSock.sin_addr), host->h_addr, host->h_length);
    // connect the socket
    if (connect(mySocket, (sockaddr *)&serverSock, sizeof(serverSock)) == -1)
    {
        cerr << "can't connect" << endl;
        WSACleanup();
        return -1;
    }
    // send data
    if ((size = send(mySocket, text.c_str(), text.size() + 1, 0)) == -1)
    {
        cerr << "Can't send data" << endl;
        WSACleanup();
        return -1;
    }
    cout << "sent " << size << endl;
    // recieve data
    text = "";
    while (((size = recv(mySocket, buf, BUFSIZE - 1, 0)) != 0) && (size != -1))
    {
        buf[size] = '\0';
        totalSize += size;
        text += buf;
    }
    if (size == -1)
    {
        cout << "Can't recieve data" << endl;
    }    
    // close connection
    closesocket(mySocket);
    WSACleanup();
    cout << "Accepted: " << totalSize << " bytes" << endl << "HTTP Header:" << endl << endl;
    // get http reply...
    int offset = text.find("\r\n\r\n");
    copy(text.begin(), text.begin() + offset, ostream_iterator<char>(cout,""));
    copy(text.begin() + offset, text.end(), ostream_iterator<char>(output,""));
    return 0;
}

This is done for VC++ v.6

**nmadct** · Mar 29 '07, 06:10 PM

First, you're way better off doing this in Perl than in C, as there's a huge amount of ready-made code that will do most of the work for you. I think Python also has good libraries for this.

If you want a robust way to access web pages, you might try libwww, although I've found it's not the easiest thing to use.

The easiest way to grab web pages from a C program is probably to invoke the wget or curl program to fetch the page as a file, then open that file.

As for parsing the file, that's an entirely different question. There is plenty of literature out there on writing parsers, you can search Google for it. If you're not interested in fully parsing HTML, but rather just getting some info out of the page, your job might be considerably easier.

**manontheedge** · Mar 30 '07, 05:38 PM

Code:

#include<stdio.h>
#include<winsock2.h>

#pragma comment(lib, "ws2_32.lib")
#define STRING_MAX 65536
#define MAX 8388608

char *get_http(char *targetip, int port, char *file)
 {
      WSADATA wsaData;
     WORD wVersionRequested;
     struct hostent*          target_ptr;
     struct sockaddr_in      sock;
     SOCKET MySock;


     wVersionRequested = MAKEWORD(2, 2);
     if (WSAStartup(wVersionRequested, &wsaData) < 0)
     {
             printf("################# ERROR! ###################\n");
             printf("Your ws2_32.dll is too old to use this application.    \n");
             printf("Go to microsofts web site to download the most recent \n");
             printf("version of ws2_32.dll.\n");


             WSACleanup();
             exit(1);
     }
     MySock = socket(AF_INET, SOCK_STREAM, 0);
     if(MySock==INVALID_SOCKET)
     {
             printf("Socket error!\r\n");

             closesocket(MySock);
             WSACleanup();
             exit(1);
     }
     if ((target_ptr = gethostbyname(targetip)) == NULL)
     {
             printf("Resolve of %s failed, please try again.\n", targetip);

             closesocket(MySock);
             WSACleanup();
             exit(1);
     }
     memcpy(&sock.sin_addr.s_addr, target_ptr->h_addr, target_ptr->h_length);
     sock.sin_family = AF_INET;
     sock.sin_port = htons((USHORT)port);

     if ( (connect(MySock, (struct sockaddr *)&sock, sizeof (sock) )))
     {
             printf("Couldn't connect to host.\n");

             closesocket(MySock);
             WSACleanup();
             exit(1);
     }
     char sendfile[STRING_MAX];
     strcpy(sendfile, "GET ");
     strcat(sendfile, file);
     strcat(sendfile, " HTTP/1.1 \r\n" );
     strcat(sendfile, "Host: localhost\r\n\r\n");
     if (send(MySock, sendfile, sizeof(sendfile)-1, 0) == -1)
     {
             printf("Error sending Packet\r\n");
             closesocket(MySock);
             WSACleanup();
             exit(1);
     }
     send(MySock, sendfile, sizeof(sendfile)-1, 0);


	     char *recvString = new char[MAX];
     int nret;
     nret = recv(MySock, recvString, MAX + 1, 0);


     char *output= new char[nret];
     strcpy(output, "");
     if (nret == SOCKET_ERROR)
     {
             printf("Attempt to receive data FAILED. \n");
     }
     else
     {
             strncat(output, recvString, nret);
             delete [ ] recvString;
     }
     closesocket(MySock);
     WSACleanup();
     return (output);
     delete [ ] output;
 }

int main(int argc, char *argv[])
{
    int port = 80;
    char* targetip;
    
    if (argc < 2)
    {
       printf("WebGrab usage:\r\n");
       printf("%s <TargetIP> [port]\r\n", argv[0]);
       return(0);
    }
    
    targetip = argv[1];
    char* output;
    
    if(argc >= 3)
    {
       port = atoi(argv[2]);
    }
    
    if(argc >= 4)
    {
       output = get_http(targetip, port, argv[3]);
    }
    
    else
    {
       output = get_http(targetip, port, "/");
    }
    
    printf("%s", output);
    
    return(0);
}

I looked in to everything you guys recommended, and I started researching sockets. In theory it makes sense. I have some code here that is suppose to get a web page and print the data to the console. But it keeps stopping at the "Couldn't connect to host" part, and it's got me confused.

I'm loading the program in the command prompt, and there are 3 arguments, one madatory...the IP address or a fully qualified domain name. Each time I run the program, about 5-10 seconds later I get that error. I'd like to know why this is happening.

I'm actually trying to learn this stuff and how it works so I can use it more in the future, so any help or guidance with this would help. Thanks for the help so far as well.

**nmadct** · Apr 2 '07, 10:45 PM

I noticed that your "memcpy" line doesn't match the one in Jan's code. I wonder if that's causing a problem. After you've tried to connect and encountered an error, you can call WSAGetLastError () to get an error message, which might be helpful. Also, it might be helpful if you could print out the values of the arguments to connect() just before making the call, to verify what it's trying to do.

I don't know what kind of reference you're using, but this page is helpful: http://www.sockets.com/winsock.htm
(It's old, but I don't think the API has changed much since then. Things have been added but the original functions should work the same.)

**EDevMachine** · Apr 25 '07, 01:37 AM

Another option if a C++/MFC app works is good for you...

// Just returning the whoooole webpage as a string for parsing.
CString GetSourceHtml(C String theUrl)
{
// this first block does the actual work
CString szOutput;
szOutput = "";
CInternetSessio n session;
CInternetFile* file = NULL;
try
{
// try to connect to the URL
file = (CInternetFile* ) session.OpenURL (theUrl);
}
catch (CInternetExcep tion* m_pException)
{
// set file to NULL if there's an error
file = NULL;
m_pException->Delete();
}

// most of the following deals with storing the html to a file
CStdioFile dataStore;

if (file)
{

CString somecode;

// continue fetching code until there is no more
while (file->ReadString(som ecode) != NULL)
{
szOutput = szOutput + somecode;
}

file->Close();
delete file;
}

return szOutput.Trim() ;

}

http://www.edevmachine.com

parsing internet page using C

parsing internet page using C

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment