Compile flex & bison

**John Harrison** · Oct 9 '05, 03:35 PM

Re: Compile flex & bison

pavel.orehov@gm ail.com wrote:[color=blue]
> Hi,
>
> I am using flex and bizon to write HTTP parser.
> I am passing well flex and bison tools but can't compile their output.
>[/color]

[snip]
[color=blue]
>
> 4. g++ -c lex.yy.cc http_parser.tab .cpp
> http_parser.lpp : In member function `virtual int yyFlexLexer::yy lex()':
> http_parser.lpp :222: error: `yyerror' undeclared (first use this
> function)
> http_parser.lpp :222: error: (Each undeclared identifier is reported
> only once
> for each function it appears in.)
> lex.yy.cc:1174: error: `yywrap' undeclared (first use this function)
> http_parser.ypp : In function `int main()':
> http_parser.ypp :90: error: `FlexLexer' undeclared (first use this
> function)
> http_parser.ypp :90: error: (Each undeclared identifier is reported only
> once
> for each function it appears in.)
> http_parser.ypp :90: error: `lexer' undeclared (first use this function)
> http_parser.ypp :90: error: parse error before `;' token
>
> What i am doing wrong ???
>
> Thanks a lot.
>[/color]

Flex and bison are really ugly, they use confusing macros all over the
place and they don't interface cleanly with each other or with C++.

I'm not exactly sure (this isn't a felix/bison group after all) but I
would say that you need to put #include <FlexLexer.h> at the start of
http_parser.ypp . Really to fix these bugs you are going to have to dig
around in the awful code that flex and bison generate and figure out for
yourself what the causes are. Usually it will be that you have to
manually add some prototype, header file or macro to your grammar or
scanner file.

While you are trying to fix these problems it is a good idea to use the
options that prevent #line directives appearing in the generated output
(-L for flex, not sure for bison). That way at least you get the correct
line numbers for errors reported.

Also while you are getting the hang of this, it might be a better idea
to start small.

john

**pavel.orehov@gmail.com** · Oct 9 '05, 03:55 PM

Re: Compile flex & bison

Okey i have fixed dome things and now i have a linking problem.
It can't find yylex function.

========= Flex =========
%{
#include <iostream>
#include "http_parser.ta b.hpp"

extern int yywrap(void);

//#define ECHO yyerror("Parse error.")

%}

%x REQUEST
%x AFTER_METHOD
%x RESP_VERSION
%x AFTER_RESP_VERS ION
%x RESP_CODE
%x AFTER_RESP_CODE
%x RESP_MSG
%x AFTER_RESP_MSG
%x REQ_HOST
%x REQ_AFTER_HOST
%x REQ_PORT_DELIMI TER
%x REQ_PORT
%x REQ_PATH
%x AFTER_REQ_PATH
%x REQ_BEFORE_VERS ION
%x REQ_VERSION
%x AFTER_REQ_VERSI ON
%x HEADER_NAME
%x HEADER_DELIMITE R
%x HEADER_VALUE
%x AFTER_HEADER_VA LUE

WS [\x20\t]+
CRLF (\r\n|\r|\n)
TOKEN [^\x80-\xFF\x7F\x00-\x1F\x20\x22\r\ n\t]+
LETTER [a-zA-Z]
HEADER_NAME [-_a-zA-Z]+
DIGIT [0-9]
PROTOCOL (http:\/\/|ftp:\/\/)
HOST [^\x2F\x3A\x80-\xFF\x7F\x00-\x1F\x20\x22\r\ n\t]+
PORT {DIGIT}+
METHOD {LETTER}+

%%

{METHOD}/{WS} {
BEGIN AFTER_METHOD;

std::cout << "REQUEST\n" ;
std::cout << "METHOD='" << YYText() << "'\n";

return T_REQ_METHOD;
}

<AFTER_METHOD>{ WS} {
BEGIN REQUEST;
}

"HTTP/" {
BEGIN RESP_VERSION;

std::cout << "RESPONSE\n ";
}

<RESP_VERSION>{ DIGIT}+\.{DIGIT }+/{WS} {
BEGIN AFTER_RESP_VERS ION;

std::cout << "RESP_VERSION=' " << YYText() << "'\n";

return T_RESP_VERSION;
}

<AFTER_RESP_VER SION>{WS} {
BEGIN RESP_CODE;
}

<RESP_CODE>{DIG IT}+/{WS} {
BEGIN AFTER_RESP_CODE ;

std::cout << "RESP_CODE= '" << YYText() << "'\n";

return T_RESP_CODE;
}

<AFTER_RESP_COD E>{WS} {
BEGIN RESP_MSG;
}

<RESP_MSG>{TOKE N}/{CRLF} {
BEGIN AFTER_RESP_MSG;

std::cout << "RESP_MSG=' " << YYText() << "'\n";

return T_RESP_MSG;
}

<AFTER_RESP_MSG >{CRLF} {
BEGIN HEADER_NAME;
}

<REQUEST>"/" {
BEGIN REQ_PATH;

std::cout << "REQ_RELATIVE_1 \n";
}

<REQUEST>"/"{WS} {
BEGIN REQ_BEFORE_VERS ION;

std::cout << "REQ_RELATIVE_2 \n";
}

<REQUEST>{PROTO COL} {
BEGIN REQ_HOST;

std::cout << "REQ_ABSOLUTE_1 \n";
std::cout << "PROTOCOL=' " << YYText() << "'\n";

return T_REQ_PROTOCOL;
}

<REQ_HOST>{HOST }/: {
BEGIN REQ_PORT_DELIMI TER;

std::cout << "REQ_HOST_1 ='" << YYText() << "'\n";

return T_REQ_HOST;
}

<REQ_HOST>{HOST }/\/ {
BEGIN REQ_PATH;

std::cout << "REQ_HOST_2 ='" << YYText() << "'\n";

return T_REQ_HOST;
}

<REQ_HOST>{HOST }/{WS} {
BEGIN REQ_AFTER_HOST;

std::cout << "REQ_HOST_3 ='" << YYText() << "'\n";

return T_REQ_HOST;
}

<REQ_AFTER_HOST >{WS} {
BEGIN REQ_BEFORE_VERS ION;
}

<REQ_PORT_DELIM ITER>":" {
BEGIN REQ_PORT;
}

<REQ_PORT>{PORT }/\/ {
BEGIN REQ_PATH;

std::cout << "REQ_PORT_1 ='" << YYText() << "'\n";

return T_REQ_PORT;
}

<REQ_PORT>{PORT }/{WS} {
BEGIN REQ_BEFORE_VERS ION;

std::cout << "REQ_PORT_2 ='" << YYText() << "'\n";

return T_REQ_PORT;
}

REQ_PATH>{TOKEN }/{WS} {
BEGIN AFTER_REQ_PATH;

std::cout << "REQ_PATH=' " << YYText() << "'\n";

return T_REQ_PATH;
}

<AFTER_REQ_PATH >{WS} {
BEGIN REQ_BEFORE_VERS ION;
}

<REQ_BEFORE_VER SION>"HTTP/" {
BEGIN REQ_VERSION;
}

<REQ_VERSION>{D IGIT}+\.{DIGIT} +/{CRLF} {
BEGIN AFTER_REQ_VERSI ON;

std::cout << "REQ_VERSIO N='" << YYText() << "'\n";

return T_REQ_VERSION;
}

<AFTER_REQ_VERS ION>{CRLF} {
BEGIN HEADER_NAME;
}

<HEADER_NAME>{H EADER_NAME}+/:{WS} {
BEGIN HEADER_DELIMITE R;

std::cout << "HEADER_NAM E='" << YYText() << "'\n";

return T_HEADER_NAME;
}

<HEADER_DELIMIT ER>:{WS} {
BEGIN HEADER_VALUE;
}

<HEADER_VALUE >[^\r\n]*/{CRLF} {
BEGIN AFTER_HEADER_VA LUE;

std::cout << "HEADER_VALUE=' " << YYText() << "'\n";

return T_HEADER_VALUE;
}

<AFTER_HEADER_V ALUE>{CRLF} {
BEGIN HEADER_NAME;
}

<AFTER_HEADER_V ALUE>{CRLF}{CRL F} { ; }

%%

int yywrap(void) {
return 1;
}

========= Bison =========

%{
#include <iostream>
#include <FlexLexer.h>

extern void yyerror(char *);

#ifdef __cplusplus
extern "C"
{
int yylex(void);
}
#endif

%}

%token T_REQ_METHOD
%token T_REQ_PROTOCOL
%token T_REQ_HOST
%token T_REQ_PORT
%token T_REQ_PATH
%token T_REQ_VERSION

%token T_RESP_VERSION
%token T_RESP_CODE
%token T_RESP_MSG

%token T_HEADER_NAME
%token T_HEADER_VALUE

%%

program:
| request
| response
;

request:
/* Absolute requests */
T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_PORT T_REQ_PATH
T_REQ_VERSION headers {
std::cout << "ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_PORT T_REQ_PATH T_REQ_VERSION headers\n";
}
|
T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_PATH T_REQ_VERSION
headers {
std::cout << "ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_PATH T_REQ_VERSION headers\n";
}
|
T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_VERSION headers {
std::cout << "ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_VERSION headers\n";
}
|
/* Relative requests */
T_REQ_METHOD T_REQ_PATH T_REQ_VERSION headers {
std::cout << "REL => T_REQ_METHOD T_REQ_PATH T_REQ_VERSION
headers\n";
}
|
T_REQ_METHOD T_REQ_VERSION headers {
std::cout << "REL => T_REQ_METHOD T_REQ_VERSION headers\n";
}
;

response:
T_RESP_VERSION T_RESP_CODE T_RESP_MSG headers {
std::cout << "T_RESP_VER SION T_RESP_CODE T_RESP_MSG headers\n";
}
;

headers:
headers header
|
;

header:
T_HEADER_NAME T_HEADER_VALUE {
std::cout << "T_HEADER_N AME T_HEADER_VALUE\ n";
}
;

%%

void yyerror(char * msg)
{
std::cout << "ERROR: " << msg << std::endl;
}

int main(void)
{
FlexLexer* lexer = new yyFlexLexer;
while(lexer->yylex() != 0)
;

return 0;
}

========= Makefile =========

APP = http_parser
SOURCE = resp_header
BISON_OPTIONS = -dv
FLEX_OPTIONS = --c++

$(APP): lex.yy.o $(APP).tab.o
g++ -o $(APP) $(APP).tab.o lex.yy.o # -lm -lfl
./$(APP) < $(SOURCE)

lex.yy.o: lex.yy.cc
g++ -c lex.yy.cc

$(APP).tab.o: $(APP).tab.cpp
g++ -c $(APP).tab.cpp

lex.yy.cc: $(APP).tab.hpp $(APP).tab.cpp
flex $(FLEX_OPTIONS) $(APP).lpp

$(APP).tab.hpp:
bison $(BISON_OPTIONS ) $(APP).ypp

$(APP).tab.cpp:
bison $(BISON_OPTIONS ) $(APP).ypp

run:
./$(APP) < $(SOURCE)

clean:
-rm *.tab.* lex.yy.* *.o *~ *.output

=========== Output =========

bison -dv http_parser.ypp
flex --c++ http_parser.lpp
g++ -c lex.yy.cc
g++ -c http_parser.tab .cpp
g++ -o http_parser http_parser.tab .o lex.yy.o # -lm -lfl
http_parser.tab .o(.text+0x261) : In function `yyparse()':
: undefined reference to `yylex'
collect2: ld returned 1 exit status
make: *** [http_parser] Error 1

**John Harrison** · Oct 9 '05, 04:05 PM

Re: Compile flex & bison

pavel.orehov@gm ail.com wrote:[color=blue]
> Okey i have fixed dome things and now i have a linking problem.
> It can't find yylex function.
>[/color]

[snip]
[color=blue]
>
> int main(void)
> {
> FlexLexer* lexer = new yyFlexLexer;
> while(lexer->yylex() != 0)
> ;
>
> return 0;
> }
>[/color]

Two problems I can see. Firstly in main you should be calling yyparse
not lexer->yylex. The parser calls the scanner, in your code you are
calling the scanner only so no parsing is going to happen.

Normally the parser calls a function called yylex to do the scanning.
Because you've chosen to use a C++ scanner the yylex function doesn't
get generated. I guess the easiest way round this is to write your own
version of yylex that calls lexer->yylex. You can use the %lex-param
directive to add extra paramters to yylex. Something like that anyway, I
am not an expert.

john

**Jacques Labuschagne** · Oct 9 '05, 10:05 PM

Re: Compile flex & bison

pavel.orehov@gm ail.com wrote:[color=blue]
> Okey i have fixed dome things and now i have a linking problem.
> It can't find yylex function.
>[/color]
<snip>[color=blue]
> ========= Makefile =========
>
> APP = http_parser
> SOURCE = resp_header
> BISON_OPTIONS = -dv
> FLEX_OPTIONS = --c++
>
> $(APP): lex.yy.o $(APP).tab.o
> g++ -o $(APP) $(APP).tab.o lex.yy.o # -lm -lfl
> ./$(APP) < $(SOURCE)[/color]

You're not linking the Flex library. '#' is the comment character.

Jacques.

**pavel.orehov@gmail.com** · Oct 10 '05, 07:25 AM

Re: Compile flex & bison

> You're not linking the Flex library. '#' is the comment character.

I don't know what these flugs for, but i have tried with them, the same
problem.
[color=blue]
>I guess the easiest way round this is to write your own version of yylex that calls lexer->yylex.[/color]

It worked, but now i only lex works without yacc/bison grammar.

How should i start grammar checker ?

============ Changed yacc/bison ============
%{
#include <iostream>
#include <FlexLexer.h>

#ifdef __cplusplus
extern "C"
{
int yylex(void);
void yyerror(char *);
int yyparse(void);
}
#endif

%}

%token T_REQ_METHOD
%token T_REQ_PROTOCOL
%token T_REQ_HOST
%token T_REQ_PORT
%token T_REQ_PATH
%token T_REQ_VERSION

%token T_RESP_VERSION
%token T_RESP_CODE
%token T_RESP_MSG

%token T_HEADER_NAME
%token T_HEADER_VALUE

%%

program:
| request
| response
;

request:
/* Absolute requests */
T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_PORT T_REQ_PATH
T_REQ_VERSION headers {
std::cout << "ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_PORT T_REQ_PATH T_REQ_VERSION headers\n";
}
|
T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_PATH T_REQ_VERSION
headers {
std::cout << "ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_PATH T_REQ_VERSION headers\n";
}
|
T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_VERSION headers {
std::cout << "ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_VERSION headers\n";
}
|
/* Relative requests */
T_REQ_METHOD T_REQ_PATH T_REQ_VERSION headers {
std::cout << "REL => T_REQ_METHOD T_REQ_PATH T_REQ_VERSION
headers\n";
}
|
T_REQ_METHOD T_REQ_VERSION headers {
std::cout << "REL => T_REQ_METHOD T_REQ_VERSION headers\n";
}
;

response:
T_RESP_VERSION T_RESP_CODE T_RESP_MSG headers {
std::cout << "T_RESP_VER SION T_RESP_CODE T_RESP_MSG headers\n";
}
;

headers:
headers header
|
;

header:
T_HEADER_NAME T_HEADER_VALUE {
std::cout << "T_HEADER_N AME T_HEADER_VALUE\ n";
}
;

%%

void yyerror(char * msg)
{
std::cout << "ERROR: " << msg << std::endl;
}

int yylex(void)
{
FlexLexer* lexer = new yyFlexLexer;
while(lexer->yylex() != 0)
;
}

int main(void)
{
yyparse();
return 0;
}

**pavel.orehov@gmail.com** · Oct 10 '05, 12:45 PM

Re: Compile flex & bison

Should i implement yylex by myself and return tokens match to
dictionary rules ???

**pavel.orehov@gmail.com** · Oct 10 '05, 02:45 PM

Re: Compile flex & bison

Okey, i found the way to solve it.

I have declared global lexer in bison input file:

yyFlexLexer lexer;

and

int yylex(void)
{
return lexer.yylex();
}

It works fine with grammar.

Now my next question is how can i call yyparse() not from main in beson
input file ?
I mean i want other external main.cpp file and call yyparse() from
there. What should i include in this file to know yyparse() function ?

**John Harrison** · Oct 10 '05, 04:55 PM

Re: Compile flex & bison

pavel.orehov@gm ail.com wrote:[color=blue]
> Okey, i found the way to solve it.
>
> I have declared global lexer in bison input file:
>
> yyFlexLexer lexer;
>
> and
>
> int yylex(void)
> {
> return lexer.yylex();
> }
>
> It works fine with grammar.
>
> Now my next question is how can i call yyparse() not from main in beson
> input file ?
> I mean i want other external main.cpp file and call yyparse() from
> there. What should i include in this file to know yyparse() function ?
>[/color]

I don't think there is anything to include. I think you just write

extern int yyparse();

int main()
{
yyparse();
}

john

Compile flex & bison

Compile flex & bison

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment