Toolchain and Free Software Testing

Table of Contents

Introduction and Motivation

Many years ago, I got interested in VAX computers, which at that time had a rough Linux port running. However, GCC continued to be a problem: Every now and then, something broke the VAX backend, so I started to do regular GCC builds to catch breakages as early as possible.

Over time, this testing effort grew. A lot.

The setup now uses the Laminar CI system to run all these different build jobs.

These days, I build Binutils/GAS/GDB for quite a lot of targets, as well as GCC. This GCC is then used to build as many Linux defconfigs as possible. In addition to my initial VAX/Linux interests, I started to do NetBSD cross builds from a Linux host, as well as building NetBSD from within a NetBSD amd64 VM. I also integrated glibc's script to build full cross toolchains including glibc.

To match by initial VAX interests, all upstreamed Open SIMH are also built. The microvax3900 simulator is then used to boot the latest VAX NetBSD install ISO and install it (using expect) fully automated. These NetBSD VAX instances are then used to build a few pkgsrc packages.

Along with these large projects, a few smaller code based are built: elfutils are a competing alternative to parts of Binutils. And the Open SIMH project is accompanied by also builting an emerging CMake integration, as well as the SIMH tools containing various filesystem conversion utilities. Also, the IANA timezone code is built, as I once had issues with it after being imported to NetBSD and breaking a NetBSD build, as well as all the sample configurations documented for crosstool-NG and buildroot.

tl;dr: Build Results



As many buildroot builds are failing (libstdc++ linking issues while building cmake), I just killed these builds.

Also, the regenerated Macroassembler repo is moved away from GitHub to my own hosting. It can be found at

There's is Alfred Arnold's Macroassembler AS around which I wanted to include into the regular CI builds. As Alfred only published tarballs, I created a GIT repo from available historic tarballs. (Please notice that this repo will be moved to later on as I don't want to forcefully "participate" in Github's 2FA.)

Building the Macroassembler AS was a great experience: Even with -Wall -Wextra -Werror -pedantic if simply built without any issues. Well done!

That's nice: Centralized storing away build artifacts, which instantly allows me to store build results for several builds of the same commit hash. (The GIT commit hash was the main filename difference when storing build results.) By simply adding the build number (now as it is using a single function across all builds) I can store more build results, which in turn can be used to compare two consecutive builds of the same sources for reproducibility.

Round #7 finished to build. This includes some Binutils bisecting as a small upstream patch to ld broke command line handling, which led to broken Linux kernel builds for the sh and hppa targets.

Some small patch went in into NetBSD to fix date strings in the INSTALL documents. These showed differences for NetBSD- vs. Linux-based builds due to differences in date's command line options.

That fixes almost all INSTLAL differences, except paper size. While it seems that the Postscript version should be in US Letter format, a freshly installed NetBSD current will produce A4 output, while a Linux cross-build will actually generate US Letter format... This isn't fully understood by now, need to further investigate on this.

After using a Qemu-based NetBSD 9 VM for the longest time to do native NetBSD builds from within a NetBSD, I've now reworked my setup to use the latest self-built NetBSD amd64 Install ISO to auto-install a -current Qemu VM (which, with throwaway overlays, can be spawned several times to do a number of builds in parallel.)

The next full Round #6 report is available.

The next full Round #5 report is available.

SIMH is a well-known machine simulator, which I use a lot for simulating VAX hardware. However, there is another such simulator for a KA630 around, by Mouse, at git:// That started as a project running on just one machine, but Mouse is actively adding support for current stock NetBSD as well as Linux.

This simulator still has some glitches which prevents current NetBSD from booting, but I guess we can get that sorted out. Once done, it should be able to run a regular NetBSD install ISO, have network support (most notably PCAP and TAP support, but also BPF and a homegrown TCP-encapsulation format) as well as disk support (including writeable overlays, somewhat similar to SIMH's or Qemu's VHD overlay support.)

While I'm still playing with NetBSD VAX and an automated setup of pbulk builders, I also added a CI job for building retro-fuse ( It's a neat project that adopts original (!) operating system sources to a FUSE adapter, carefully updating the old sources. So this should be compatible with old systems, even bug-compatible. As a bonus, access to old filesystems through the retro-fuse programs is even verified using the original old operating system on a SIMH simulated PDP/11. (The SIMH PDP/11 binary actually also originates from our CI builds.)

As compile-testing works without any major issues, my current plan is to put together a VAX VM (OpenSIMH) and prepare it as a pbulk builder using an amd64-based distcc host. Something like that was already done by other people (eg. see in combination with general pbulk ( and distcc ( docs), but my approach is to script it to a point where I can easily reproduce such a setup for further targets.

Round #4 is done. Took a bit longer, as I did some more stuff in between.

Building Ada as a cross-compiler has its own issues: It actually requires an up-to-date Ada compiler on the host system. With the new round started, a locally built compiler will generally be used instead of one from the gcc-snapshot package. That should give us a compiler that's suitable to build Ada. Let's see if that works for all targets, it was only tested for a aarch64-linux build.

Round #3 Results are out! This is the last round to contain OpenSIMH builds with cmake, and the first round to also include builds using the buildcross script. It provides some more coverage, but is expected to also produce a good number of failed targets as it also builds ancient GCC configurations that will only build using older GCC versions.

Round #2 Results are out!

Round #1 Results are out!

Finished a new central script to start emulators (Qemu, GXemul and Open SIMH) in a common way to

That should make it quite easy to start different NetBSD install ISOs on different simulated hardware, and start pkgsrc builders afterwards.

Installation is driven by an expect script, which should quite useable to also install NetBSD on real hardware. Maybe something like RaSCSI/PiSCSI could help here.

As all the stuff seems to build correctly, I started to look into an issue with the GCC testsuite. As you'd see, the tests weren't really attempted at all. Turns out, the local user ID needs to resolve to a name. Too bad that doesn't work when you run with numeric uid/gid supplied in a Docker container. Username lookup now works and the testsuite attempts to run all the tests locally. Unsuccessful of course. Next step: Figure out how to properly configure Dejagnu for all the simulators. Fortunately, there is a How to test GCC on a simulator page around. Thanks a lot for it!

With the first full documented round of builds, all Buildroot configurations where attempted as well. With 268 samples, Buildroot took about 1/5 of the whole compile time. Of those, 254 (= 95%) were successful, while 14 (= 5%) failed. Appropriate tickets were opened.

Instead of triggering certain jobs, I started to build a full round of all jobs. Thus there's now a Round #0 Results overview page.

When cross-building NetBSD-amd64 from a Linux-amd64 setup, we'd observe an interesting behavior: During the build process, nbmandoc is built which is used to convert man pages to HTML. That's invoked multiple times during the build, until ... zlib gets build. Suddenly, nbmandoc fails to start with a dynamic linker problem: it cannot load The GNU/linux system uses, but the requested library,, is the NetBSD target libc!

What caused this? For all the CI builds, I have a little shell fragment which allows me to choose between different compilers. It sets $CC, $CXX and $LD_LIBRARY_PATH to use system GCC, some different clang versions or the GCC from Debian's gcc-snapshot package. Usually, the gcc-snapshot configuration is used.

While setting $CC and $CXX is as trivial as expected, setting up $LD_LIBRARY_PATH turned out to be more interesting than expected. The initial approach was
export LD_LIBRARY_PATH="/usr/lib/gcc-snapshot/lib:${LD_LIBRARY_PATH}"
which was ment to have the required path set and preserve whatever was in $LD_LIBRARY_PATH. This results in an empty path component which I expected the dynamic linker to ignore. However, this was not the case! As I learned, an empty path component in $PATH (man bash) as well as in $LD_LIBRARY_PATH (man evaluates to the current directory. As the NetBSD libz was also built for amd64, it was pulled in by from $PWD, but of course it couldn't find NetBSD's as the destdir was in no search path.

To correctly extend a variable like $PATH or $LD_LIBRARY_PATH, use this:
export LD_LIBRARY_PATH="/usr/lib/gcc-snapshot/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

Initial support for buildroot was added. That's 268 new Laminar jobs and as these aren't exactly quick to build (at least on my available hardware), they'll probably be queued at irregular intervals. Rough estimation: This will add four days of build time.

With the new crosstool-NC and buildroot jobs, Laminar now manages about 2300 FOSS configurations!

All 115 crosstool-NG configurations were scheduled. Of these initial builds, 111 were successful and 4 failed:

Configuration Issue
crosstoolng-loongarch64-unknown-linux-gnu Invalid configuration. Run 'ct-ng menuconfig' and check which options select INVALID_CONFIGURATION.
crosstoolng-x86_64-multilib-linux-uclibc,moxie-unknown-moxiebox Installing ISL for hostBuilding ISL:
/tmp/x86_64-multilib-linux-uclibc/bin/../lib/gcc/x86_64-multilib-linux-uclibc/12.2.0/../../../../x86_64-multilib-linux-uclibc/bin/ld.bfd: failed to set dynamic section sizes: bad value
crosstoolng-x86_64-multilib-linux-uclibc,powerpc-unknown-elf Installing ISL for hostBuilding ISL:
/tmp/x86_64-multilib-linux-uclibc/bin/../lib/gcc/x86_64-multilib-linux-uclibc/12.2.0/../../../../x86_64-multilib-linux-uclibc/bin/ld.bfd: failed to set dynamic section sizes: bad value
crosstoolng-x86_64-pc-linux-gnu,arm-picolibc-eabi Required toolchain x86_64-pc-linux-gnu does not exist, though x86_64-unknown-linux-gnu exists. So it's possibly enough to rename this configuration.

New generator job for building all of crosstool-NG's sample configurations. Final first builds should show up during the next days.

Current Issues

Component Type Description Discussion Status
NetBSD RB sandpoint/powerpc Bug Don't embed build-time timestamps. misc/57915 Open
NetBSD RB evbarm/earmv4eb Bug Don't embed build-time timestamps. misc/57914 Open
NetBSD RB acorn32/earmv4 Bug Don't embed build-time timestamps. misc/57913 Open
NetBSD RB hppa/hppa Bug Don't embed build-time timestamps. misc/57911 Open
CT-NG riscv64-multilib-elf Bug GCC still uses Python 2 for multilib parsing 2039 Open
GCC local (et al.?) Regression Header dependency issue with generated file 110248, Email Open
GCC mips-linux et al. Regression New warning in build/ Email Open
Buildroot hifive_unleashed_defconfig Bug Linker errors in opensbi-0.9 #15658 Open
CT-NG i686-w64-mingw32,nios2-spico-elf and x86_64-w64-mingw32,arm-cortexa9_neon-linux-gnueabihf Bug Issue linking GDB with the Windows cross-compiler. #1982 Open
NetBSD reproducible builds Bug Not as reproducible as thought: The source filenames may get into binaries (__FILE__), which went undetected as the test script ( does all builds it compares to each other in the very same directory. Also, with -g in CFLAGS it seems the full $PWD ends up in the binaries as well. All this came up while getting reproducible VAX builds. Some more work is underway to actually find these differences in reproducible builds. #57466 Open
GCC (Rust fork) all Regression error: '*(unsigned int*)((char*)&old_module + offsetof(Rust::Optional,Rust::Optional::))' may be used uninitialized [-Werror=maybe-uninitialized] #2242 Open
Buildroot pc_x86_64_bios_defconfig Bug Missing header file for Linux kernel build #15586 Open
CT-NG arm-multilib-linux-uclibcgnueabi Bug gettimeofday.c:(.text+0x38): undefined reference to `_dl_errno' #1959 Open
GXemul Wishlist Allow to configure MAC addresses for emulated NICs. Open
Laminar Wishlist Improve performance with large databases / long output #192 Open
Buildroot pcengines_apu2_defconfig Bug Possibly truncated output in snprintf() #15451 Open
Buildroot snps_arc700_axs101_defconfig Bug msort.c:200: Error: opcode 'dmb' not supported for target arc700 #15446 Open
Buildroot raspberrypi3_qt5we_defconfig Bug Your Buildroot configuration needs a compiler capable of building 32 bits binaries. #15441 Open
Buildroot qemu_s390x_defconfig Bug FS image too small #15436 Open
Buildroot orangepi_one_plus_defconfig Bug Error: arch/arm/dts/.sun50i-h6-orangepi-one-plus.dtb.pre.tmp:151.1-10 syntax error #15431 Open
Buildroot socrates_cyclone5_defconfig Bug bison: missing operand #15426 Open
Buildroot nitrogen8mp_defconfig, nitrogen8mn_defconfig, nitrogen8mm_defconfig, nitrogen8m_defconfig Bug u-boot: error: command 'swig' failed: No such file or directory #15416 Open
Buildroot beaglebone_qt5_defconfig Bug ERROR: file(am335x-bonegreen-wireless.dtb): No such file or directory #15406 Open
Buildroot beaglebone_defconfig, grinn_chiliboard_defconfig, nitrogen6x_defconfig, nitrogen7_defconfig Bug Some sample configurations require libssl-dev to be built/installed, but that's missing in their config. #15396 Open
GXemul Wishlist No support for tap networking using Linux. There are Github forks around with the needed small fixes, should probably also be included upstream. #1032471, Open
Laminar Wishlist Better runtime calculation to see what to expect for the next build #191 Open
GAS (breaking GCC) mipsisa64sr71k-elf Bug During Modula-2 components:
Assembler messages:
Error: bad value (sr71k) for default CPU
Internal error in mips_after_parse_args at config/tc-mips.c:15291
#30197 Open
GCC m32c-elf Bug/Regression ICE while building Modula-2 components #109026 #83670 Open
GDB mipsisa64sb1-elf Regression error: unknown conversion type character 'v' in format [-Werror=format=] and error: format '%ld' expects argument of type 'long int', but argument 9 has type 'unsigned int' [-Werror=format=] #29439 Open
Laminar Bug Pages count wrong. #190 Open
Laminar Wishlist Display running, unfinished runs in the graph displayed at the job's page #188 Open
GCC epiphany-elf, epiphany-elfOPT-with-stack-offset=16 Bug Unrecognizable insn (internal compiler error: in extract_insn, at triggered during Modula-2 build #108928 Open
GCC ICEs, breaking Linux ia64-bigsur_defconfig, ia64-generic_defconfig, ia64-gensparse_defconfig, ia64-tiger_defconfig and ia64-zx1_defconfig Bug ICE: qsort checking failed (error: qsort comparator non-negative on sorted output: 1) (Breaks Linux ia64 builds) #87281 Open
GCC nds32le-linux, nds32be-elf, nds32le-elf Bug error: array subscript 2 is above array bounds of 'rtx_def* [2]' #100837 Open

NetBSD RB acorn32/earmv4, cats/earmv4, epoc32/earmv4, hpcarm/earmv4, iyonix/earm, shark/earmv4, zaurus/earm Bug Don't embed build-time timestamps. misc/57912 Open
NetBSD RB amd64/x86_64 Bug Don't embed build-time timestamps. misc/57910 Fixed
NetBSD RB hp300/m68k Bug Don't embed build-time timestamps. misc/57909 Fixed
NetBSD RB bebox/powerpc Bug Don't embed build-time timestamps. misc/57906 Fixed
Binutils (ld: Regression in command line option handling breaks sh and hppa Linux kernel builds. Regression A small patch to ld broke Linux kernel builds for most sh and all hppa configurations. 28910 Fixed
retro-fuse Bug Regex doesn't match Python's current strict rules 8 Fixed
GCC local (et al.?) Regression Header dependency issue with generated file 110248, Email Fixed
Binutils ld alpha-dec-vms and alpha64-dec-vms Regression Access to non-existing flags struct member #30568 Fixed
Buildcross all Feature Request Allow to build from GIT HEAD branches for all configurations (even those known-broken with current code.) #2 Implemented
GCC (powerpc(64)-linux) breaking linux-powerpc (ie. linux-powerpc-adder875_defconfig Bug/Regression ICE with asm goto #110420 Fixed
Buildroot mx53loco_defconfig Bug Missing dependency on OpenSSL #15661 Fixed
Buildroot engicam_imx6qdl_icore_qt5_defconfig, imx6-sabresd_qt5_defconfig Bug glmark2: Invalid argument 'function_name' in test #15411 Fixed
CT-NG sparc-leon-linux-uclibc Bug /tmp/crosstoolng/.build/sparc-leon-linux-uclibc/src/gcc/gcc/graphite-isl-ast-to-gimple.c:349:3: error: 'isl_val_free' was not declared in this scope; did you mean 'isl_vec_free'? #1962 Fixed
Buildroot qemu_ppc_mpc8544ds_defconfig Bug Patch for Linux kernel does no longer apply #15581 Fixed
GCC mcore-elf Bug error: 'prev_addr' may be used uninitialized in this function [-Werror=maybe-uninitialized] #100332 Fixed
Buildroot qemu_aarch64_ebbr_defconfig Bug Missing kernel tarball #15421 Fixed
GCC avr-elf, pru-elf, rl78-elf Regression New warning after memchr optimization #106900 Fixed Bug With --full-gcc, all builds break (probably due to no --disable-gcov) on eg. glibcbot-alpha-linux-gnu/21/src/gcc/libgcc/libgcov.h:49:10: fatal error: sys/mman.h: No such file or directory #100289, Patchwork Fixed
simh-cmake Regression libm isn't being linked, leading to linking error due to missing math symbols. Comment Fixed
GCC (all targets) Bug IP_END handling for asm goto #107997 Fixed
GCC lm32-uclinux Bug ../.././gcc/config/lm32/uclinux-elf.h:70: error: "LINK_GCC_C_SEQUENCE_SPEC" redefined [-Werror] #100268 Fixed
GCC microblaze-linux Bug RTX may be used uninitialized in this function #100836 Fixed
GCC (Rust fork) msp430-elfbare, msp430-elf Regression ICEs during Rust selftest for the msp430 targets. #1713, #1973 Fixed
Compiler Explorer Wishlist Include the vax-netbsdelf GCC into Compiler Explorer #4783 Implemented
Laminar Bug Crashes with multiple browser tabs and large (multi-gigabyte) database files #185 Fixed
CT-NG Bug Wrong HOST toolchain configured #1909 and #1911 Fixed
NetBSD pmax-mipsel Bug Error: symbol type "gnu_indirect_function" is not supported by MIPS targets #57245 Fixed
NetBSD vax-vax Bug Support 512 MB RAM in SIMH VAX #56758 Fixed
NetBSD nnetbsd-zaurus-earm / netbsd-zaurus-earm Bug cc1: error: target CPU does not support interworking #56238 Fixed
NetBSD hpcmips-mipsel Bug nbmakefs: Writing inode 1936 (dir), bytes 0 + 512: No space left on device #56477 Fixed
NetBSD amd64-x86_64 Bug Target lib (libc, via libz) pulled in accidentally on host program #57265 Closed (CI setup problem)
elfutils Bug Testcases fail with current Debian versions when dbgsym packages for libc (probably x86_84 + x32 for multiarch) are not installed. #29176 Fixed
Laminar Bug Grouped jobs not displayed when there's a job without a run #189 Already fixed upstream
CT-NG Bug Add unzip (Build-Depends) and python3 (Depends) to ./debian/control #1905 Fixed
Laminar Wishlist Add a link from a run page to it's job page #184 Implemented
GCC powerpc-linux, powerpc64-linux Regression ICE when building Linux's arch/powerpc/kernel/align.c (asm goto) #108095 Fixed
GCC csky-linux-gnu Bug config/csky/csky.h:402:13: error: comparison of unsigned expression in '>= 0' is always true [-Werror=type-limits] #100603 Fixed
GCC (Rust fork) pdp11-aout Regression pdp11_gen_int_label() not declared (header issue) #1864 Fixed
GCC (Rust fork) powerpc-ibm-aix7.1, powerpc-ibm-aix7.2 Regression rs6000_xcoff_strip_dollar() not declared (header issue) #1819 Fixed
GCC (Rust fork) pdp11-aout Regression Warning in #1714 Invalid (build compiler issue)
NetBSD mips(el) Regression Mips(el) targets broken after Binutils Update (no IFUNC for MIPS). #57245 Fixed

Components Under Test and Components Used as Test Framework

Component Home Docs Source Bugs
Laminar CI Docs Github Issues
GNU Compiler Collection Manual Git Bugzilla
GNU Compiler Collection (Rust fork) Github Issues
Binutils, gas Docs Sourceware Bugzilla
GDB Docs Sourceware Bugzilla
glibc Manual, single page, PDF Sourceware Bugzilla
Linux gitweb
crosstool-NG Docs Github Issues
elfutils Sourceware Bugzilla
NetBSD Guide (as PDF), Reproducible Builds src Query, Submit, Summary
pkgsrc (not yet) pkgsrc guide (as PDF) pkgsrc
Open SIMH Github Issues
Open SIMH Tools Github Issues
GXemul Docs Download Email
Tools for unusual tape formats ("tuhstools") Github Issues
IANA Timezone Database / Tools Docs, BCP 175 Github Issues
Compiler Explorer How it works Github Issues
diffoscope Try Diffoscope Salsa Issues
Buildroot Manual git Bugzilla
Buildcross Github Issues

Testing Efforts by Others

Location Tesing GCC, GCC (Rust fork), Binutils, GDB, glibc, elfutils NetBSD GCC GCC GDB

2023-07-15 Jan-Benedict Glaw <> or +49-152-33822499.