SCons build tool speed

[ted]

How does the speed of the Scons build tool compare with Ant? Right now with
out Ant builds take around an hour. Hoping to speed that up.

TIA,
Ted

[Christopher De Vries]

Re: SCons build tool speed

On Sat, Feb 12, 2005 at 07:16:02PM +0000, ted wrote:[color=blue]
> How does the speed of the Scons build tool compare with Ant?[/color]

I would recommend asking this question on users@scons.tig ris.org , but my
impressions is that most of the time is probably spent in the compiler. If you
are working on a java project you could try switching from javac to jikes and
that might improve your time, though it has been a while since I used jikes.

Chris

[Peter Hansen]

Re: SCons build tool speed

ted wrote:[color=blue]
> How does the speed of the Scons build tool compare with Ant? Right now with
> out Ant builds take around an hour. Hoping to speed that up.[/color]

Don't tools like Scons, Ant, and for that matter "make" just
execute other programs? So that 99% of the time is consumed
external to the Scons, Ant, or make process itself? Why
do you think the speed of the build tool is of any significance
compared to the time for things like compilers and linkers
to execute?

-Peter

[Mike Meyer]

Re: SCons build tool speed

Peter Hansen <peter@engcorp. com> writes:
[color=blue]
> ted wrote:[color=green]
>> How does the speed of the Scons build tool compare with Ant? Right
>> now with out Ant builds take around an hour. Hoping to speed that up.[/color]
>
> Don't tools like Scons, Ant, and for that matter "make" just
> execute other programs? So that 99% of the time is consumed
> external to the Scons, Ant, or make process itself? Why
> do you think the speed of the build tool is of any significance
> compared to the time for things like compilers and linkers
> to execute?[/color]

Actually, these tools do a number of things oter than run other
programs. For instance, they parse input files to construct a build
tree, walk the build tree figuring out what needs to be rebuilt, and
only *then* do they get around to executing those other
programs. Assuming that the work prior to running those other programs
is "neglible" can lead to some truly atrocious build-time behavior. In
fact, one major advantage that ant/scons/jam/etc. have over Make is
that they avoid the defacto standard "recursive make" build
system. See <URL:
http://www.pcug.org.au/~millerp/rmch...cons-harm.html > for
details on that.

<mike
--
Mike Meyer <mwm@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Neil Hodgson]

Re: SCons build tool speed

ted:
[color=blue]
> How does the speed of the Scons build tool compare with Ant?
> Right now with out Ant builds take around an hour. Hoping to
> speed that up.[/color]

Scons emphasises accuracy over speed and is normally a little slower than
other build tools although still fast enough for most purposes. One cause of
slowness is that it reads the source files, traces include files and
checksums them all rather than relying on file times. There are some things
you can do to speed it up:

http://www.scons.org/cgi-bin/wiki/GoFastButton

Neil

[PA]

Re: SCons build tool speed


On Feb 13, 2005, at 12:20, Neil Hodgson wrote:
[color=blue]
> http://www.scons.org/cgi-bin/wiki/GoFastButton[/color]

Out of curiosity, why does scons uses MD5 by default? Is that not, er,
somewhat heavy handed for most practical purpose?

Would not some sort of lightweight CRC be "good enough" to start with?

http://samba.anu.edu.au/rsync/tech_report/node3.html

Cheers

--
PA, Onnay Equitursay

http://alt.textdrive.com/

[knight@baldmt.com]

Re: SCons build tool speed

Hi Neil, ted, et al.--
[color=blue][color=green]
> > How does the speed of the Scons build tool compare with Ant?
> > Right now with out Ant builds take around an hour. Hoping to
> > speed that up.[/color]
>
> Scons emphasises accuracy over speed and is normally a little
> slower than other build tools although still fast enough for most
> purposes. One cause of slowness is that it reads the source files,
> traces include files and checksums them all rather than relying
> on file times. There are some things you can do to speed it up:
> http://www.scons.org/cgi-bin/wiki/GoFastButton[/color]

In anticipation of the next release of SCons, we've been doing
a *lot* of work on profiling the performance and eliminating
bottlenecks.

It turns out that scanning the source files and performing the MD5
calculation is not the dominant factor that we've all assumed it is.
There were some other inefficiencies in some of our algorithms that
were much more significant, including some unnecessary disk scans,
recomputing the same dependencies for every target in a list of
targets generated by one command, and repeated just-in-time creation
of some internal objects that could be created just once and cached.

That said, SCons will never be as fast as Make, because it *is*
doing more for you under the covers. Although the next version
won't necessarily speed up every configuration (the bottlenecks
are extremely configuration dependent), it should be a significant
improvement for many configurations out there.

--SK

[knight@baldmt.com]

Re: SCons build tool speed

Hi Peter--
[color=blue][color=green]
> > How does the speed of the Scons build tool compare with
> > Ant? Right now with out Ant builds take around an hour. Hoping
> > to speed that up.[/color]
>
> Don't tools like Scons, Ant, and for that matter "make" just
> execute other programs? So that 99% of the time is consumed
> external to the Scons, Ant, or make process itself? Why
> do you think the speed of the build tool is of any significance
> compared to the time for things like compilers and linkers
> to execute?[/color]

You're right that build times are dominated by the external commands
when rebuilds occur. If nothing needs to be rebuilt, though, the
wall-clock time is obviously dominated by how long it takes for
the build tool to decide that. A build tool that doesn't decide
whether or not targets are up-to-date *and* dispatch the commnds
to rebuild them reasonably quickly and efficiently doesn't do a
good job of serving its users.

--SK

[Peter Hansen]

Re: SCons build tool speed

knight@baldmt.c om wrote:[color=blue]
> You're right that build times are dominated by the external commands
> when rebuilds occur. If nothing needs to be rebuilt, though, the
> wall-clock time is obviously dominated by how long it takes for
> the build tool to decide that. A build tool that doesn't decide
> whether or not targets are up-to-date *and* dispatch the commnds
> to rebuild them reasonably quickly and efficiently doesn't do a
> good job of serving its users.[/color]

That answer, combined with Mike's response pointing out
that tools more sophisticated than basic "make" actually
can delve into the source and identify the dependencies,
removes my confusion. Thanks. :)

-Peter