RE: Cygwin build system SOOOO SLOOOWWWW ???

[ericblake at comcast dot net (Eric Blake)]

> we've been struggling with speeding up our build process, too, so I'd
> like to share some ideas - yet I'm not so certain whether you'll learn
> from me or I'll learn from you: Hopefully both.

Thanks for a nice email, with quite a bit of useful and accurate info!

> - "strace" - though I haven't ever used it, but from what I know
>   this will definitely give you an answer - maybe two much of it ;-)

Yes, strace will tell you everything that cygwin is doing under
the hood, but it slows down normal execution and is not always
easy to parse to see what the time-killers are.

> 
> > 2) Has anyone else experienced speed problems with Cygwin?  Has
> > anybody else felt that Cygwin has gotten slower over the last
> > year or so?  Are there any guidelines or "tricks" for getting
> > Cygwin to run faster? 

Actually, I haven't noticed any slowdowns, and cgf has been
adding things to snapshots that try to speed up some cases.

> 
> a) Forking is more expensive in Windoze.
> On Unix, especially in make environments, you'll often start new
> processes as you're going - and often you'll not even notice. Google
> for "bash tricks" on how to fork less often.
> Hint: Don't use "sed" in `backticks` just for simple string
> replacements.
> Much of this can be done in make or bash directly.
> Look at the changes you made - maybe you thought it's more elegant?

That is true.  CVS libtool is gaining quite a bit of speed by deliberately
using bash builtins in place of fork()/exec() sequences.

> c) /bin/sh is now bash, which is now dynamically linked.
> Up until a few months ago, /bin/sh has been "ash", a smaller, but
> less powerfull shell. This has been replaced by bash, to reduce the
> traffic of repeated questions along the lines of "why does my shell
> act different than on linux" (where /bin/sh is bash on most
> distributions).
> If I understood the traffic on this list correctly, bash is now
> dynamically
> linked, which might have an impact on starting it - I can't tell.

It shouldn't have much impact - the reason we made the switch
from ash to bash is because no one could produce a test case
where bash was significantly slower.

> Hint: Don't start bash so often. Create fewer processes, but if you
> must, see if you gain by using ash explicitely instead of bash.
> 
> To the gurus - is the following correct?
> `echo blub` starts one process, `echo blub | sed -e 's/b/x/g'`
> starts three: "echo", "sed", and "bash" to implement the pipe.

Not quite - echo is a bash builtin, so it is bash and not echo.  But
you are right that it is 1 fork vs. 3 forks and 1 exec.

Here's where strace can be handy:
strace bash -c 'your command here' | grep '^Program name'
will show you how many forks and execs took place (execs
reuse a pid, forks create a new one).

> 
> d) Beware of lazy evaluation.
> Look at this construct:
> CFLAGS=$(shell find . -type d -name include)
> Read "info make" on setting variables and find out about the
> difference between "=" and ":=". The above will run the find
> again for every single call to the compiler. Along with the
> issues about forking and reading directories and small files,
> this can make a difference of *ages*.
> Hint: See whether you can use less variables, use ":=" more often,
> etc. - and don't use "$(shell ...)" anyway, as stated in a).
> Rather, pre-compute makefiles with all the data hardcoded, using
> ":=".

Definately true.  There are efficient ways of using make, and
there are slick constructs, and the two don't always overlap.

> 
> e) Reading lots of small files seems more expensive on Windoze.
> I don't know about your Makefiles, but traditionally, makefiles are
> spread across project directories (for build hierarchies), and
> makedepend
> creates even more of that. For one of our applications, I roughly
> calculated that make needs to open, read, and parse well over a
> thousand files (not counting the source or objects or any such thing,
> just the makefiles), just for telling you that all the targets are
> up to date.

Building monolithic makefiles at the top level instead of
recursing into subdirectory makes can speed up builds,
as well as give make better dependency information.
Google for 'recursive make considered harmful'.

--
Eric Blake



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/