Hi,
https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ is big news.
Besides that, this month, I spent a bigger part of my time to get r-b-patches into our SLFO project that will be the basis for SLE-16 and Leap-16.
There are only ~10 issues left.
The harder ones are with emacs and PGO. And maybe rpm.
Meanwhile, the automatic rb-checker for Factory submissions does help to fix some issues before they even enter Factory.
It helps to reduce regressions.
I temporarily paused the verification builds because Factory was lacking a new rpm-macro for rpm-4.20 causing extra variations in mtime.
It will be back soon - after the next big rebuild.
I'm also happy to see that this month, at least 4 other people (plus upstream devs) contributed patches for reproducibility issues in openSUSE. If we can keep that level, we should reach 100% before my previous estimate of year 2030.
Here are the autogenerated bits:
last month's status:
https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/PPDVJBF22DYFYI6BT7ONGHQLHUUJU7W3/
Last months' reproducible builds project updates (including my work):
https://reproducible-builds.org/reports/2025-01/
I uploaded https://rb.zq1.de/compare.factory-20250228/ today
https://rb.zq1.de/spec/glossar.txt explains the meaning of below values:
total-packages: 15993 (+90)
build-tried: 15940 (+94)
build-failed: 13 (+9)
build-n-a: 173 (-1)
build-succeeded: 15754 (+86)
build-official-failed+na: 102 (+60)
build-compare-failed: 173 (-2)
build-compare-succeeded: 15581 (+88)
verify-failed: 226 (-13)
verified-semi-reproducible: 15401 (-10)
verified-bit-identical: 15144 (-85)
bit-by-bit-identical: 15476 (+90)
not-bit-by-bit-identical: 278 (-4)
not-bit-by-bit-identicalcheck: 278 (-4)
https://rb.zq1.de/compare.factory-20250228/graph.png
shows the change over time
https://rb.zq1.de/compare.factory-20250228/unreproduciblerings.txt
lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages,
2 were in ring0
23 were in ring1
That makes it 25/4043 => 0.62 %
which is below the overall average of
173/15754 => 1.10 %
278/15754 => 1.76 % of packages are not perfectly reproducible
package notes:
==> ./abseil-cpp/.rb.notes <==
=> SR 1237987 1237960 = https://bugzilla.opensuse.org/show_bug.cgi?id=1235867 race?
1-core-VM still varies ; nothing helps
==> ./apache-ivy/.rb.notes <==
=> SR 1206032 jar javadoc html => strip-nd
==> ./avr-libc/.rb.notes <==
minor .a
strange varying hardlinks from fdupes
==> ./bcc/.rb.notes <==
=> SR 1245394 https://github.com/iovisor/bcc/pull/5213
https://build.opensuse.org/request/show/1245248 by wfrisch bcc lua
==> ./build/.rb.notes <==
nachbau: https://github.com/openSUSE/obs-build/pull/1054
==> ./chezmoi/.rb.notes <==
FTBFS -j1
go1.19 parallelism
==> ./ck/.rb.notes <==
= https://github.com/concurrencykit/ck/issues/243
=> https://github.com/concurrencykit/ck/pull/184 FTBFS-j1
==> ./colord/.rb.notes <==
= https://bugzilla.opensuse.org/show_bug.cgi?id=1237156 = https://github.com/hughsie/colord/issues/174 https://bugzilla.opensuse.org/show_bug.cgi?id=1217747 .icc only-CPU AVX2 ; toolchain from lcms2?
==> ./crash/.rb.notes <==
= https://bugzilla.opensuse.org/show_bug.cgi?id=1230281 parallelism, race
==> ./curl/.rb.notes <==
FTBFS-2033
found range good=2087143195 bad=2087145119
==> ./deepin-gir-generator/.rb.notes <==
debuginfo=--debuginfo multibuildrbkall ; nothing helps.
/usr/share/go/1.17/contrib/src/pkg.deepin.io/gir/gudev-1.0/gudev.gen.c 1f61e81a27f2d77e4fcbc3f34de5f22c0be5960ba523fd1e9375a62257626e2b 0
==> ./difftastic/.rb.notes <==
=> https://github.com/Wilfred/difftastic/pull/813 race rust/llvm
==> ./dunst/.rb.notes <==
=> https://github.com/dunst-project/dunst/pull/1435 date
--- old /usr/bin/dunst (objdump)
==> ./eclipse-egit/.rb.notes <==
=> SR 1239889 strip-nondeterminism
in /usr/share/eclipse/droplets/egit-egit/plugins/org.eclipse.egit.core.junit.source_5.11.0.202103091610-r.jar
==> ./erlang/.rb.notes <==
FTBFS-j1
https://bugzilla.opensuse.org/show_bug.cgi?id=1120947 date+time
==> ./exaile/.rb.notes <==
=> https://github.com/exaile/exaile/pull/956 parallelism/race
in exaile.desktop
==> ./flacon/.rb.notes <==
parallelism+lto
in debuginfo
==> ./forgejo-guardian/.rb.notes <==
rust/llvm toolchain? nothing helps
==> ./gawk/.rb.notes <==
=> SR 1243466 1240443 PGO/bug
==> ./ghc-wai-app-static/.rb.notes <==
ghc toolchain? debuginfo=--debuginfo ; nothing helps ; minor
==> ./ghc-yaml/.rb.notes <==
ghc toolchain? debuginfo=--debuginfo ; nothing helps
==> ./ghc/.rb.notes <==
ordering issue in /usr/lib64/ghc-9.8.3/lib/x86_64-linux-ghc-9.8.3/libHSCabal-3.10.3.0-8d80-ghc9.8.3.so ; nothing helps
==> ./ghostscript/.rb.notes <==
=> SR 1243309 1243016 date in .h
+ https://bugs.ghostscript.com/show_bug.cgi?id=696765
==> ./gstreamer-plugins-rs/.rb.notes <==
= https://bugzilla.opensuse.org/show_bug.cgi?id=1237097 = https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/599 date ; rust/llvm asm diff
variations in /usr/lib64/pkgconfig/gstwebrtchttp.pc
==> ./hdf5/.rb.notes <==
=> SR 1242841 :gnu-hpc debuginfo date+hostname
+++ new//usr/src/debug/hdf5-1.12.3/src/H5lib_settings.c 2024-05-13 11:41:09.000000000 +0000
==> ./helm/.rb.notes <==
FTBFS-2028-04-05
2032-08-21 SSL
==> ./intel-graphics-compiler/.rb.notes <==
= https://github.com/intel/intel-graphics-compiler/issues/359 ASLR build has variations in some (likely generated) source files: /home/abuild/rpmbuild/BUILD/build/IGC/VectorCompiler/lib/BiF/VCBuiltins64.cpp /home/abuild/rpmbuild/BUILD/build/IGC/Release/bif/embedder/OCLBiFImpl__igc_bif_BIFBC_122.cpp
==> ./ioquake3/.rb.notes <==
=> https://github.com/ioquake/ioq3/pull/704 debugsource: date+time
==> ./joker/.rb.notes <==
=> SR 1240852 1240514 #=> https://github.com/candid82/joker/commit/d2817bb18cd9cc0274460113a97dc87f2613d1ad
#= https://github.com/candid82/joker/issues/491 hash random order
==> ./julia/.rb.notes <==
stuck in -j1 build => workaround -j3
==> ./kbd/.rb.notes <==
2024-12 https://github.com/legionus/kbd/commit/eebaa3b69efd9e3d218f3436dc43ff3340020ef5
==> ./libmfx-gen/.rb.notes <==
vmtype=--vm-type=podman rbk
uname -r inmediasdk_build_info
==> ./liburing/.rb.notes <==
FTBFS-j1?
==> ./llamacpp/.rb.notes <==
#=> https://github.com/ggerganov/llama.cpp/pull/11366 CPU
#= https://github.com/ggerganov/llama.cpp/issues/11306 random order
==> ./lout/.rb.notes <==
.pdf from ghostscript/gs, toolchain
==> ./lowfi/.rb.notes <==
= https://github.com/servo/html5ever/issues/573 toolchain markup5ever generated.rs
/usr/bin/lowfi buildid
==> ./lua-language-server/.rb.notes <==
FTBFS-2038
--build-opt=--vm-custom-opt=-rtc base=2042-01-25T00:00:00 failed
==> ./lychee/.rb.notes <==
=> SR 1247017 = https://github.com/servo/html5ever/issues/573 = https://github.com/lycheeverse/lychee/issues/1632 random (rust), regression?
==> ./mage/.rb.notes <==
=> SR 1246041 => https://github.com/magefile/mage/pull/474 date+time ; golang
from .goreleaser.yml: ldflags: -s -w -X github.com/magefile/mage/mage.timestamp={{.Date}} -X github.com/magefile/mage/mage.commitHash={{.Commit}} -X github.com/magefile/mage/mage.gitTag={{.Version}}
==> ./mayavi/.rb.notes <==
FTBFS-j1 / other?
tvtk_classes.zip varies (mtimes?)
==> ./messagelib/.rb.notes <==
parallelism in /usr/share/doc/qt6/KPim6MessageViewer.qch - toolchain?
==> ./mingw64-zlib/.rb.notes <==
varies with debuginfo
UNIX epoch: - needs binutils update?
==> ./neovim/.rb.notes <==
= https://github.com/neovim/neovim/issues/26387 - maybe toolchain https://github.com/LuaJIT/LuaJIT/issues/1008
= https://github.com/neovim/neovim/issues/32429 FTBFS-2038
==> ./nlopt/.rb.notes <==
=> SR 1244447 nlopt jar mtime
==> ./nushell/.rb.notes <==
=> SR 1247358 1247347 toolchain rust HashSet
==> ./nvidia-modprobe/.rb.notes <==
=> SR 1239771 date in /usr/share/man/man1/nvidia-modprobe.1.gz
==> ./ocaml-extlib/.rb.notes <==
parallelism in debuginfo=--debuginfo multibuildrbkall
==> ./palo/.rb.notes <==
=> SR 1247223 1247180 => patch sent per email to Helge Deller ; date in man + binary
--- old /usr/sbin/palo (objdump)
==> ./pgvector/.rb.notes <==
=> SR 1240185 1240183 https://github.com/pgvector/pgvector/pull/764 = https://bugzilla.opensuse.org/show_bug.cgi?id=1236351 CPU -march=native
==> ./pixelorama/.rb.notes <==
timestamp + ordering + random
==> ./postgresql17/.rb.notes <==
= https://www.postgresql.org/message-id/ba8e1bc0-8a99-45b7-8397-3f2e94415e03%40suse.de FTBFS-2024-11-03 - solved by postgresql17-jsonb_jsonpath.patch
found range good=1730573413 bad=1730576847
==> ./python-Django4/.rb.notes <==
=> SR 1240318 FTBFS-2038
==> ./python-aioquic/.rb.notes <==
= https://github.com/aiortc/aioquic/issues/557 FTBFS-2028
==> ./python-numpy/.rb.notes <==
=> SR 1245658 1245435 random .pyc in RECORD + mesonpy toolchain random
==> ./python-psygnal/.rb.notes <==
FTBFS-j1
==> ./python-python-datamatrix/.rb.notes <==
= https://bugzilla.opensuse.org/show_bug.cgi?id=1236437 FTBFS disk<RAM
==> ./python-rdflib/.rb.notes <==
FTBFS-j1 :doc (pinged packager in Slack)
==> ./python-tuf/.rb.notes <==
FTBFS-2030-01-01
found range good=1893454059 bad=1893455984
==> ./python-webauthn/.rb.notes <==
=> SR 1242191 = https://github.com/duo-labs/py_webauthn/issues/243 FTBFS-2025-02-02
found range good=1738490879 bad=1738492804
==> ./python-wxPython/.rb.notes <==
FTBFS-j1
[ 4372s] _____________________________ xml_Tests.test_xml2 ______________________________
==> ./python3-pyside6/.rb.notes <==
= https://bugreports.qt.io/browse/PYSIDE-2508 ASLR order?
_smp_build_ncpus embedded in .src.rpm https://github.com/rpm-software-management/rpm/issues/2343
==> ./qore-ssh2-module/.rb.notes <==
--- binaries.nachbau/qore-ssh2-module.spec 2024-05-26 11:58:33.000000000 +0000
+++ binaries/qore-ssh2-module.spec 2024-05-26 11:58:33.000000000 +0000
==> ./qrq/.rb.notes <==
=> SR 1246722 date+time from `date`
--- old /usr/bin/qrq (objdump)
==> ./river/.rb.notes <==
= https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo from CPU
==> ./scx/.rb.notes <==
date? rust/llvm LTO
varying order in /home/abuild/rpmbuild/BUILD/scx-1.0.9/build/scheds/c/scx_sdt.p/scx_sdt.bpf.skel.h
==> ./sdcc/.rb.notes <==
= https://sourceforge.net/p/gputils/bugs/328/ gputils toolchain-time + ghostscript toolchain
/usr/share/sdcc/lib/pic14/libm.lib
==> ./sequoia-sq/.rb.notes <==
=> https://gitlab.com/sequoia-pgp/sequoia-sq/-/merge_requests/554 FTBFS-2026-01-31
= https://gitlab.com/sequoia-pgp/sequoia-sq/-/issues/535 FTBFS-2026-10-05
==> ./stacktile/.rb.notes <==
= https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo
==> ./starship/.rb.notes <==
rust ; maybe some low-entropy (filesys) issue left
=> https://github.com/starship/starship/pull/5352 filesys readdir, rust
==> ./topgrade/.rb.notes <==
rust/llvm from https://github.com/topgrade-rs/topgrade/pull/807 ?
==> ./tucnak/.rb.notes <==
=> SR 1247238 = https://bugzilla.opensuse.org/show_bug.cgi?id=1237211 parallelism-race
missing entries in /usr/lib/tucnak/tucnak.d
==> ./uhd/.rb.notes <==
=> SR 1243555 date in debugsource
==> ./uki/.rb.notes <==
pesign
==> ./warewulf4/.rb.notes <==
=> SR 1248768 by cgoll ; embedded CPU-core count
==> ./waylock/.rb.notes <==
= https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo, CPU
==> ./wayprompt/.rb.notes <==
= https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo
--- old /usr/lib/debug/usr/bin/wayprompt.debug (objdump)
==> ./xmlgraphics-fop/.rb.notes <==
= https://bugzilla.opensuse.org/show_bug.cgi?id=1237094 date+random
=> SR 866261
==> ./yazi/.rb.notes <==
= https://github.com/lukaslueg/built/issues/80 parallelism/core-count with debuginfo
=> https://github.com/xiph/rav1e/pull/3412
==> ./zig/.rb.notes <==
= https://github.com/ziglang/zig/issues/22839 ASLR in debuginfo
=> SR 1244717 1241564 = https://github.com/ziglang/zig/issues/22663 zig toolchain, CPU-dependent path in debuginfo
==> ./zls/.rb.notes <==
=> https://build.opensuse.org/request/show/1241564 = https://github.com/zigtools/zls/issues/2138 CPU / zig toolchain ; random path in debuginfo=--debuginfo multibuildrbkall
-/usr/src/debug/zls-0.13.0/zig-cache/o/dd3075db6d19db75877b8d14c5f6fc11/version_data_0.13.0.zig 0 (none) 100644 root root 0 4294967295