Re: More crosstool-0.42-glibc-2.4-gcc-4.1.0-nptl

["Michael K. Edwards" <m dot k dot edwards at gmail dot com>]

On 5/10/06, Robert Schwebel <robert@schwebel.de> wrote:
> On Wed, May 10, 2006 at 05:40:53PM -0700, Michael K. Edwards wrote:
> > You do have the option of building on a fast x86 (or amd64) box,
> > running ARM native compiler binaries under an ARM sarge chroot using
> > qemu. You should get the same results as compiling on a real ARM,
> > only much faster -- and it works for software that doesn't have
> > cross-compile-friendly build procedures.  See my earlier post
> > regarding mkxdeb.
>
> Hmm, I'm not really happy with that approach, it goes into the same
> direction as for example scratchbox, just without real hardware. If
> cross compiling is broken in tools we are using, shouldn't we fix them
> instead of adding just another hack to make it work for us? IMHO we
> should work towards fixing the upstream packets. That's the policy
> we've followed with PTXdist for the last years.

Don't get me wrong -- working cross-compile build procedures are a
good thing.  But I wouldn't really want to cross-compile, say, Perl,
or even the Python extension that glues to libxml2; too much of the
build procedure relies on being able to run binaries immediately as
they are compiled.  I am a lot more comfortable with qemu than with
scratchbox, because the only sleight of hand involved is at the opcode
and syscall level, and there are absolutely no host binaries involved
other than qemu itself (and the libraries it calls and the host
kernel, if you like).

Timing aside, there should be no differences between the result of
compiling under an ARM chroot and qemu and the result of compiling on
a "real" ARM.  Except, of course, that you don't have to find an ARM
you can hook up to a RAID array, cram a gig of memory into, and run at
GHz speeds in order to compile, say, TAO in less than 24 hours.  I
don't need an ORB on my embedded system today, but I may well next
year -- and I prefer not to disqualify code bases with heavy tool
requirements (C++ templates, anyone?) and extensive compile-time test
harnesses just because it's impractical to cross-compile them.

I might add that it's about time embedded Linux projects started using
grown-up software packaging techniques (I like dpkg and apt; YMMV) and
that mkxdeb makes it almost as easy to bootstrap a rootfs image for an
ARM on an x86 dev host as it is to build a native rootfs with
debootstrap.  (You do, of course, need a Debian repository for your
arch.)  I've done it the hard way too, and I'm doing it the hard way
again right now (not least because there is no big-endian ARM port in
sarge or AFAIK sid either) -- but as far as I'm concerned that's only
the first stage in a scorched-earth bootstrap procedure.

You build your crosstools, you build your minimal chroot and a
Canadian cross, then you qemu-chroot in and build a fully native
toolchain.  Instantiate a dpkg database with the host's dpkg and use
equivs to fake up dpkg entries for your toolchain.  Build your
Required packages and real packages for build-essential, then
construct a fresh qemu-chroot, this time with debootstrap and your
pile o' packages.  If you're paranoid, rebuild all your packages in
this chroot and debootstrap afresh -- it's your first chance to test
that your glibc built the Debian Way really works.

Then start layering on applications, without worrying about whether
they cross-compile easily.  And if you find yourself slinging XML in
C, ask yourself why you're not doing it in Python instead (or Java if
you must).  Don't settle for "I can't figure out how to cross-compile
libxml2's Python bindings" (or "I can't build against the J2ME class
library and still run Junit tests on the dev host, and modifying other
people's Ant scripts makes me go cross-eyed").

Now you have a set of independently testable, modularly packaged
software components from which you can build both development and
production target images.  That's worth having even if you drop the
dpkg database on the target and still rip-and-replace flash contents
on every field upgrade.  And if you still use the cross-compiler to
build a kernel, stripped glibc, and busybox for the production target,
they're the only things you have to test without a complete set of
diagnostic tools in hand.  Remote GDB stubs are all very well, but
when I am troubleshooting complex software misbehavior I want strace
and ltrace and Perl (and oprofile and valgrind when I can get them)
right there on the target.

Let me digress from building under emulation and talk about testing
under emulation.  In my experience, most "interesting" bugs are race
conditions and resource starvation under unanticipated load conditions
-- often exposed by an incremental change in toolchain or hardware
capabilities.  Unless your friendly neighbor with the time machine
helps you test on tomorrow's hardware today, qemu on a fast-ish system
is probably your best tool for exercising code under a variety of
different resource conditions.

Think of every design assumption that might have gone into the
embedded system, and violate it!  Run bonnie concurrently to slow I/Os
to a crawl.  Configure netfilter to drop some packets randomly, or
corrupt their TCP checksums, or do whatever else takes you out of your
network fast path.  Write a little program that copies some data from
here to there and then calls sched_yield, endlessly -- and run it at
real-time priority outside qemu, so that it blows the D-cache at every
schedule point in the target code.  Heck, add a sched_yield in qemu
every time it proxies a syscall.

How many SNMP agents can take that kind of abuse and still present a
coherent snapshot of interface counters across a series of getnext
requests?  Hint: if your stress testing is limited to flooding a black
box with dummy frames, you probably won't find out until your customer
calls to complain.  And for the large class of problems that are best
solved by throwing resources at them, it helps to know which resource
is oversubscribed -- cranking the processor clock won't help if it's
the TLB flush penalty that's killing you.

> > I haven't gotten around to doing an EABI chroot yet, but I probably
> > will soon. I need to build a basic userspace and a Canadian Cross
> > first, and will probably finish bringing up 2.6.16.x on my board
> > before monkeying further with qemu. I'd love to use PTXdist to
> > bootstrap my userspace efforts; can you hook me up with an up-to-date
> > version of PTXdist that has crosstool 0.42 integrated?
>
> Unfortunately nobody at Pengutronix has time to upgrade to 0.42 at the
> moment. But that shouldn't be a problem for you: just build your
> toolchain with the normal crosstool, download and install ptxdist-0.10.3
> and either add the toolchain to your path or use 'ptxdist toolchain'.

I'll do that.  Having evangelized the above approach, it behooves me
to follow it from beginning to end for once.  :-)

Cheers,
- Michael

--
For unsubscribe information see http://sourceware.org/lists.html#faq