#include optimization

[Karl Heinz Buchegger]

Re: #include optimization

Ramesh Natarajan wrote:[color=blue]
>
> (2) The second problem is un-necessary #includes. Over a period of time
> the
> developers have got a set of #includes that they add to the files
> without
> thinking!!
>
> for eg: consider
>
> #include<stdio. h>
> #include<sys/types.h>
> #include<sys/socket.h>
> int main(int argc,char** argv){
> printf("Hello World\n");
> }
>
> Here the includes for types and socket is unnecessary.
>
> This involves 2 file opens on this level that could be avoided.
>
> I want to fix this also.
>
> But I am unable to come out with some solution...[/color]

Hmm. Just a quick shot. The idea is:

Write a program which reads and modifies a source file. The program
drops one of the includes and afterwards starts the compiler on the
modified source. If the compiler doesn't complain, the include was
unnecesary and is permanenlty dropped. If the compiler emits an error
then the drop is undone.
Your program checks every include in this way and thus keeps only the
needed ones.

--
Karl Heinz Buchegger
kbuchegg@gascad .at

[Dan Pop]

Re: #include optimization

In <lnwtye1bb6.fsf @nuthaus.mib.or g> Keith Thompson <kst-u@mib.org> writes:
[color=blue]
>Dan.Pop@cern.c h (Dan Pop) writes:[color=green]
>> In <11e6710.040928 0435.731bebd1@p osting.google.c om>
>> ramesh25@gmail. com (Ramesh) writes:[color=darkred]
>>>I am currently maintaining a legacy code with a very very large code
>>>base. I am facing problems with C/C++ files having a lot of
>>>un-necessary #includes. On an average every C/C++ file has around
>>>150+ .h files included. I find 75% of the files unnecessary and
>>>could be removed. Considering the fact that I have a huge code base,
>>>I can't manually fix it.[/color]
>>
>> If it ain't broken, don't fix it.[/color]
>
>Unless fixing it will make future maintenance easier, which is
>probably the case here.[/color]

Why?

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Dan.Pop@ifh.de
Currently looking for a job in the European Union

[Michael Mair]

Re: #include optimization

Hi there,
[color=blue][color=green][color=darkred]
>>>>I am currently maintaining a legacy code with a very very large code
>>>>base. I am facing problems with C/C++ files having a lot of
>>>>un-necessary #includes. On an average every C/C++ file has around
>>>>150+ .h files included. I find 75% of the files unnecessary and
>>>>could be removed. Considering the fact that I have a huge code base,
>>>>I can't manually fix it.
>>>
>>>If it ain't broken, don't fix it.[/color]
>>
>>Unless fixing it will make future maintenance easier, which is
>>probably the case here.[/color]
>
> Why?[/color]

I cannot look into Keith's head but one possible reason is that
from looking at which file includes which header you get information
where to look when changing something.

Especially in huge codes with many man years of work in them and
too few man days of documentation work and me not being an expert,
this (e.g. by find . -type f -exec grep obscureheader.h \{} -print)
is -- after browsing around with tags -- one step I might
take to get a feeling for how this connects with that and whether some
people took "shortcuts" somewhere that need to be fixed.
If you basically have people including about fifty header files for
no reason in every translation unit they get into their hands so
that they do not have to know about where which functions, definitions
and so on come from, then this approach obviously is not feasible.

Just a guess.


Cheers
Michael

[John Bode]

Re: #include optimization

"E. Robert Tisdale" <E.Robert.Tisda le@jpl.nasa.gov > wrote in message news:<cjcev5$1e t$1@nntp1.jpl.n asa.gov>...[color=blue]
> Ramesh wrote:
>[color=green]
> > I am currently maintaining a legacy code with a very very large code base.
> > I am facing problems with C/C++ files having a lot of un-necessary
> > #includes.[/color]
>
> You are probably mistaken.
> Good programmers don't include header files that aren't necessary.
>[/color]

Yes they do, on occasion. It's possible that the original programmers
weren't very good. It's also possible that the headers were necessary
at one time, but over time the code was hacked beyond recognition and
the symbols in those headers are no longer being used in that source
file. Or it could be the result of a cut-and-paste fest stemming from
unclear requirements and a laughably unrealistic schedule.

I've seen this movie more than once. Hell, I'm *in* the movie.
[color=blue][color=green]
> > On an average every C/C++ file has around 150+ .h files included.
> > I find 75% of the files unnecessary and could be removed.[/color]
>
> How did you determine that?
>[/color]

By checking to see if the source file actually *uses* any of the
symbols defined in the header file maybe?
[color=blue][color=green]
> > Considering the fact that I have a huge code base,
> > I can't manually fix it.[/color]
>
> What, exactly, are you trying to fix?
>[/color]

He is trying to remove unnecessary #include directives, thereby making
the code easier to read and maintain (and maybe save some cycles
during builds).
[color=blue][color=green]
> > Are there any tools that would report un wanted .h files?[/color]
>
> Of course not. How would such a tool know
> what should and shouldn't be in any given header file?
>[/color]

The tool could scan each header file for symbols (macros, typedefs,
enums, function declarations, external variable declarations) and then
check the source file to see if any of those symbols are present. If
none of the symbols in the header are present in the source file, then
the tool can mark that header as (probably) superfluous. I don't
personally know of any tool that does that particular job, and I won't
claim that it would be easy to write one (handling nested includes
would be "fun"), but it can be done (lex/flex and yacc/bison would
probably be the best way to go about it).
[color=blue]
>
> Try this:
>[color=green]
> > cat file.h[/color]
> #ifndef GUARD_FILE_H
> #define GUARD_FILE_H 1
> #include <stdlib.h>
> #include <stdio.h>
> #include <stdint.h>
> #include <math.h>
> #include <string.h>
> #include <values.h>
> #include <time.h>
> #include <signal.h>
> #endif//GUARD_FILE_H
>[color=green]
> > cat file1.c[/color]
> #include "file.h"
>[color=green]
> > time gcc -Wall -std=c99 -pedantic -c file1.c[/color]
> 0.133u 0.049s 0:00.18 94.4% 0+0k 0+0io 0pf+0w[color=green]
> > cat file128.c[/color]
> #include "file.h"
> #include "file.h"
> .
> .
> .
> #include "file.h"
>[color=green]
> > time gcc -Wall -std=c99 -pedantic -c file128.c[/color]
> 0.144u 0.039s 0:00.19 89.4% 0+0k 0+0io 0pf+0w
>
> which shows that it doesn't take any more time to process
> a header file 128 time than it takes to process it once!
> Once the C preprocessor has read an idempotent file,
> it doesn't read it again no matter how many times it is included.[/color]

But that's not really the OP's problem, is it?

[Alan Johnson]

Re: #include optimization

Karl Heinz Buchegger wrote:[color=blue]
> Ramesh Natarajan wrote:
>[color=green]
>>(2) The second problem is un-necessary #includes. Over a period of time
>>the
>>developers have got a set of #includes that they add to the files
>>without
>>thinking!!
>>
>>for eg: consider
>>
>>#include<stdi o.h>
>>#include<sy s/types.h>
>>#include<sy s/socket.h>
>>int main(int argc,char** argv){
>>printf("Hel lo World\n");
>>}
>>
>>Here the includes for types and socket is unnecessary.
>>
>>This involves 2 file opens on this level that could be avoided.
>>
>>I want to fix this also.
>>
>>But I am unable to come out with some solution...[/color]
>
>
> Hmm. Just a quick shot. The idea is:
>
> Write a program which reads and modifies a source file. The program
> drops one of the includes and afterwards starts the compiler on the
> modified source. If the compiler doesn't complain, the include was
> unnecesary and is permanenlty dropped. If the compiler emits an error
> then the drop is undone.
> Your program checks every include in this way and thus keeps only the
> needed ones.
>[/color]

Such a program would also need to verify that the resulting executable
is identical.

Alan

[Keith Thompson]

Re: #include optimization

Dan.Pop@cern.ch (Dan Pop) writes:[color=blue]
> In <lnwtye1bb6.fsf @nuthaus.mib.or g> Keith Thompson <kst-u@mib.org> writes:[color=green]
>>Dan.Pop@cern. ch (Dan Pop) writes:[color=darkred]
>>> In <11e6710.040928 0435.731bebd1@p osting.google.c om>
>>> ramesh25@gmail. com (Ramesh) writes:
>>>>I am currently maintaining a legacy code with a very very large code
>>>>base. I am facing problems with C/C++ files having a lot of
>>>>un-necessary #includes. On an average every C/C++ file has around
>>>>150+ .h files included. I find 75% of the files unnecessary and
>>>>could be removed. Considering the fact that I have a huge code base,
>>>>I can't manually fix it.
>>>
>>> If it ain't broken, don't fix it.[/color]
>>
>>Unless fixing it will make future maintenance easier, which is
>>probably the case here.[/color]
>
> Why?[/color]

Because.

Take a look at what's been said so far. The source files have an
average of about 150 #include directives, most of which are
unnecessary. It's a "very very large code base". It seems fairly
obvious to me that the whole thing is a mess, and that cleaning it up
would make it easier to maintain. (I'm tempted to suggest the
possibility of throwing it away and starting from scratch, but that's
probably not feasible.)

Maybe it isn't really a problem, or maybe the unnecessary #includes
are such a small part of the problem that eliminating them wouldn't
really help, which is why I qualified my statement with the word
"probably". But since Ramesh specifically said that it's a problem,
and he's asking for ways to fix it, I'm not going to assume that he's
mistaken about the premise for his question.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[CBFalconer]

Re: #include optimization

Keith Thompson wrote:[color=blue]
> Dan.Pop@cern.ch (Dan Pop) writes:[color=green]
>> Keith Thompson <kst-u@mib.org> writes:[color=darkred]
>>> Dan.Pop@cern.ch (Dan Pop) writes:
>>>> ramesh25@gmail. com (Ramesh) writes:[/color][/color]
>[color=green][color=darkred]
>>>>> I am currently maintaining a legacy code with a very very large
>>>>> code base. I am facing problems with C/C++ files having a lot
>>>>> of un-necessary #includes. On an average every C/C++ file has
>>>>> around 150+ .h files included. I find 75% of the files
>>>>> unnecessary and could be removed. Considering the fact that I
>>>>> have a huge code base, I can't manually fix it.
>>>>
>>>> If it ain't broken, don't fix it.
>>>
>>> Unless fixing it will make future maintenance easier, which is
>>> probably the case here.[/color]
>>
>> Why?[/color]
>
> Because.
>
> Take a look at what's been said so far. The source files have an
> average of about 150 #include directives, most of which are
> unnecessary. It's a "very very large code base". It seems fairly
> obvious to me that the whole thing is a mess, and that cleaning it
> up would make it easier to maintain. (I'm tempted to suggest the
> possibility of throwing it away and starting from scratch, but
> that's probably not feasible.)
>
> Maybe it isn't really a problem, or maybe the unnecessary #includes
> are such a small part of the problem that eliminating them wouldn't
> really help, which is why I qualified my statement with the word
> "probably". But since Ramesh specifically said that it's a
> problem, and he's asking for ways to fix it, I'm not going to
> assume that he's mistaken about the premise for his question.[/color]

Actually it is a job that can be attacked and checked piecemeal.
Remove the (presumably) useless includes in one file by commenting
out (thus retaining line numbers) and compile it to an object
file. Do a binary compare against the original object file. If
identical, all is well. If not, investigate the causes (which may
include a compilation datestamp). This assumes that the system
will not generate external linkages on seeing only a prototype.

Another approach revolves around a cross referance, or the
interactive equivalent supplied by cscope.

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

[Peter van Merkerk]

Re: #include optimization

CBFalconer wrote:[color=blue]
> Keith Thompson wrote:
>[color=green]
>>Dan.Pop@cern. ch (Dan Pop) writes:
>>[color=darkred]
>>>Keith Thompson <kst-u@mib.org> writes:
>>>
>>>>Dan.Pop@cer n.ch (Dan Pop) writes:
>>>>
>>>>>ramesh25@g mail.com (Ramesh) writes:[/color]
>>[color=darkred]
>>>>>>I am currently maintaining a legacy code with a very very large
>>>>>>code base. I am facing problems with C/C++ files having a lot
>>>>>>of un-necessary #includes. On an average every C/C++ file has
>>>>>>around 150+ .h files included. I find 75% of the files
>>>>>>unnecessa ry and could be removed. Considering the fact that I
>>>>>>have a huge code base, I can't manually fix it.
>>>>>
>>>>>If it ain't broken, don't fix it.
>>>>
>>>>Unless fixing it will make future maintenance easier, which is
>>>>probably the case here.
>>>
>>>Why?[/color]
>>
>>Because.
>>
>>Take a look at what's been said so far. The source files have an
>>average of about 150 #include directives, most of which are
>>unnecessary . It's a "very very large code base". It seems fairly
>>obvious to me that the whole thing is a mess, and that cleaning it
>>up would make it easier to maintain. (I'm tempted to suggest the
>>possibility of throwing it away and starting from scratch, but
>>that's probably not feasible.)
>>
>>Maybe it isn't really a problem, or maybe the unnecessary #includes
>>are such a small part of the problem that eliminating them wouldn't
>>really help, which is why I qualified my statement with the word
>>"probably". But since Ramesh specifically said that it's a
>>problem, and he's asking for ways to fix it, I'm not going to
>>assume that he's mistaken about the premise for his question.[/color]
>
>
> Actually it is a job that can be attacked and checked piecemeal.
> Remove the (presumably) useless includes in one file by commenting
> out (thus retaining line numbers) and compile it to an object
> file. Do a binary compare against the original object file. If
> identical, all is well.[/color]

Even if binary compare shows that there are differences all may be well
too. I learned this the hard way; compiling the same code twice with a
certain compiler yielded different binaries. Apparently the compiler
wrote the build date or something like that into the binary. When debug
information is stored in object files this test may also fail for the
wrong reasons.

--
Peter van Merkerk
peter.van.merke rk(at)dse.nl

[Karl Heinz Buchegger]

Re: #include optimization

Alan Johnson wrote:[color=blue]
>[color=green]
> > Hmm. Just a quick shot. The idea is:
> >
> > Write a program which reads and modifies a source file. The program
> > drops one of the includes and afterwards starts the compiler on the
> > modified source. If the compiler doesn't complain, the include was
> > unnecesary and is permanenlty dropped. If the compiler emits an error
> > then the drop is undone.
> > Your program checks every include in this way and thus keeps only the
> > needed ones.
> >[/color]
>
> Such a program would also need to verify that the resulting executable
> is identical.[/color]

Ah. right.
#undef xxx
#define xxx something_else
would else be unnoticed

--
Karl Heinz Buchegger
kbuchegg@gascad .at

[CBFalconer]

Re: #include optimization

Peter van Merkerk wrote:[color=blue]
> CBFalconer wrote:[color=green]
>>[/color]
>... snip ...[color=green]
>>
>> Actually it is a job that can be attacked and checked piecemeal.
>> Remove the (presumably) useless includes in one file by commenting
>> out (thus retaining line numbers) and compile it to an object
>> file. Do a binary compare against the original object file. If
>> identical, all is well.[/color]
>
> Even if binary compare shows that there are differences all may be well
> too. I learned this the hard way; compiling the same code twice with a
> certain compiler yielded different binaries. Apparently the compiler
> wrote the build date or something like that into the binary. When debug
> information is stored in object files this test may also fail for the
> wrong reasons.[/color]

Why did you snip the last sentence or two from my paragraph, which
mentioned precisely this problem?

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

[Michael Mair]

Re: #include optimization

>>>Actually it is a job that can be attacked and checked piecemeal.[color=blue][color=green][color=darkred]
>>>Remove the (presumably) useless includes in one file by commenting
>>>out (thus retaining line numbers) and compile it to an object
>>>file. Do a binary compare against the original object file. If
>>>identical, all is well.[/color]
>>
>>Even if binary compare shows that there are differences all may be well
>>too. I learned this the hard way; compiling the same code twice with a
>>certain compiler yielded different binaries. Apparently the compiler
>>wrote the build date or something like that into the binary. When debug
>>information is stored in object files this test may also fail for the
>>wrong reasons.[/color]
>
> Why did you snip the last sentence or two from my paragraph, which
> mentioned precisely this problem?[/color]

Probably because you gave a solution to the problem without stating
the problem...
To be honest, I did not really get what you were aiming at, either.

For completeness, here is the left-out part:
[color=blue][color=green][color=darkred]
>>> Another approach revolves around a cross referance, or the
>>> interactive equivalent supplied by cscope.[/color][/color][/color]

Cheers,
Michael

[Dan Pop]

Re: #include optimization

In <lnekkkg9po.fsf @nuthaus.mib.or g> Keith Thompson <kst-u@mib.org> writes:
[color=blue]
>Dan.Pop@cern.c h (Dan Pop) writes:[color=green]
>> In <lnwtye1bb6.fsf @nuthaus.mib.or g> Keith Thompson <kst-u@mib.org> writes:[color=darkred]
>>>Dan.Pop@cern .ch (Dan Pop) writes:
>>>> In <11e6710.040928 0435.731bebd1@p osting.google.c om>
>>>> ramesh25@gmail. com (Ramesh) writes:
>>>>>I am currently maintaining a legacy code with a very very large code
>>>>>base. I am facing problems with C/C++ files having a lot of
>>>>>un-necessary #includes. On an average every C/C++ file has around
>>>>>150+ .h files included. I find 75% of the files unnecessary and
>>>>>could be removed. Considering the fact that I have a huge code base,
>>>>>I can't manually fix it.
>>>>
>>>> If it ain't broken, don't fix it.
>>>
>>>Unless fixing it will make future maintenance easier, which is
>>>probably the case here.[/color]
>>
>> Why?[/color]
>
>Because.
>
>Take a look at what's been said so far. The source files have an
>average of about 150 #include directives, most of which are
>unnecessary. It's a "very very large code base". It seems fairly
>obvious to me that the whole thing is a mess, and that cleaning it up
>would make it easier to maintain. (I'm tempted to suggest the
>possibility of throwing it away and starting from scratch, but that's
>probably not feasible.)[/color]

One could merge all the 150 headers in a single header and you'll see
a single header being included instead of 150. Including that header
will also provide some unnecessary definitions and declarations, but this
is no different from including <stdlib.h> and getting more declarations
than your application actually needs (when was the last time you included
<stdlib.h> and used everything declared within?).

So, the fact that the source files include 150 headers, not all of them
necessary to each source file, is not a problem in itself. If the
maintainer is annoyed by seeing such a bunch of includes everywhere,
he can trivially write a new header, that includes all the application
headers, and include only this header in each source file.
[color=blue]
>Maybe it isn't really a problem, or maybe the unnecessary #includes
>are such a small part of the problem that eliminating them wouldn't
>really help, which is why I qualified my statement with the word
>"probably". But since Ramesh specifically said that it's a problem,
>and he's asking for ways to fix it, I'm not going to assume that he's
>mistaken about the premise for his question.[/color]

And I'm not going to believe him until provided with *concrete* examples.

I'm tempted to believe that the mistake was creating so many application
specific headers in the first place and the *right* fix would be to
drastically reduce their number. Without knowing the specifics, there
is no way of telling whether merging all of them in a single header
(e.g. by using another header that includes all of them) is the right
fix or if there is a real need for more than one application specific
header. But I'm reasonably convinced that there is no need for 150 header
files.

OTOH, I'm reasonably convinced that their existence is not causing
any maintenance problems, either. It's just that the source files
containing so many include directives look mildly annoying.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Dan.Pop@ifh.de
Currently looking for a job in the European Union

[CBFalconer]

Re: #include optimization

Michael Mair wrote: *** And removed attributions ***[color=blue]
>[color=green][color=darkred]
>>>> Actually it is a job that can be attacked and checked piecemeal.
>>>> Remove the (presumably) useless includes in one file by commenting
>>>> out (thus retaining line numbers) and compile it to an object
>>>> file. Do a binary compare against the original object file. If
>>>> identical, all is well.
>>>
>>> Even if binary compare shows that there are differences all may be
>>> well too. I learned this the hard way; compiling the same code
>>> twice with a certain compiler yielded different binaries.
>>> Apparently the compiler wrote the build date or something like
>>> that into the binary. When debug information is stored in object
>>> files this test may also fail for the wrong reasons.[/color]
>>
>> Why did you snip the last sentence or two from my paragraph, which
>> mentioned precisely this problem?[/color]
>
> Probably because you gave a solution to the problem without stating
> the problem...
> To be honest, I did not really get what you were aiming at, either.
>
> For completeness, here is the left-out part:
>[color=green][color=darkred]
>>> Another approach revolves around a cross referance, or the
>>> interactive equivalent supplied by cscope.[/color][/color][/color]

Wrong left-out part, which was as follows, and was an integral
portion of the quoted paragraph:
[color=blue][color=green][color=darkred]
>>> identical, all is well. If not, investigate the causes (which may
>>> include a compilation datestamp). This assumes that the system
>>> will not generate external linkages on seeing only a prototype.[/color][/color][/color]

Your elimination of attributions leaves the (mistaken) impression
that I was complaining about your actions, while the actual
offender was Peter van Merkerk.

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

[Michael Mair]

Re: #include optimization

[Me messing up a discussion I did not take part in][color=blue][color=green]
>>
>>For completeness, here is the left-out part:
>>
>>[color=darkred]
>>>>Another approach revolves around a cross referance, or the
>>>>interacti ve equivalent supplied by cscope.[/color][/color]
>
>
> Wrong left-out part, which was as follows, and was an integral
> portion of the quoted paragraph:[/color]

Argh, sorry, I did so not get it :-([color=blue]
>
>[color=green][color=darkred]
>>>>identical , all is well. If not, investigate the causes (which may
>>>>include a compilation datestamp). This assumes that the system
>>>>will not generate external linkages on seeing only a prototype.[/color][/color]
>
> Your elimination of attributions leaves the (mistaken) impression
> that I was complaining about your actions, while the actual
> offender was Peter van Merkerk.[/color]

Er, right. Was a quick one when compiling my crap...
More thinking before sending next time.
Once again, sorry for messing it up!

--Michael