Hi, This month, the number of verified packages increased a lot. This has two reasons: 1. I added a 'nachbauall' script to handle multibuild packages. 2. on 2025-01-10 a lot of packages were rebuilt for the removal of python310. This helped because I use the osc build -p (prefer) option and it has limitations. e.g. if a package was built with go1.22 and I provide that old rpm, it will still use go1.23 for the build. Also prjconf changes can cause issues. Verified packages are those where I was able to produce identical or equivalent (per build-compare) packages outside of OBS. Debian and Arch do such checks using rebuilderd. https://rb.zq1.de/compare.factory/build-compare-differed-builds-nachbau.txt lists packages that could not be verified. Most of these have known issues. ghc still causes issues for verification of ~80 packages. There is some randomness added, when debuginfo is enabled. I also configured the tests to build these with debuginfo, because it makes a difference there, so the numbers will look a bit worse from that. On the positive side, I saw fewer build-failures than ever, so we get the clearest view on the current state. https://github.com/openSUSE/obs-build/pull/1047 stands out as a surprising bug that affected multiple packages, but only when trying to build them on a host with a non-UTC timezone. Another similar issue showed up, when changes entries had unparseable dates, such as "Fri Dez 11 16:35:40 UTC 2020" (notice 'Dez' instead of 'Dec') - then the result depended on the current date. https://build.opensuse.org/requests/1235463 brought a toolchain fix that helped 'rusty_v8' and there seems to have been another toolchain fix, because some other rust reproducibility issues vanished as well. Meanwhile for my Reproducible-Builds-Operating-System (RBOS) project (sponsored by a NLnet NGI0 grant), I reached a new level of reproducibility by producing bit-identical rpms in OBS, osc build and pbuild and verifying this for most packages in https://build.opensuse.org/project/show/home:bmwiedemann:reproducible:distribution:ring0 . Only multibuild packages need more scripting work here. Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/BKMFGPNWUCNLKZOWPA7GGKBERJBS4WN6/ Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2024-12/ I uploaded https://rb.zq1.de/compare.factory-20250131/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15903 (-4) build-tried: 15846 (-8) build-failed: 4 (-21) build-n-a: 174 (-1) build-succeeded: 15668 (+14) build-official-failed+na: 42 (-109) build-compare-failed: 175 (+14) build-compare-succeeded: 15493 (+0) verify-failed: 239 (-11) verified-semi-reproducible: 15411 (+311) verified-bit-identical: 15229 (+1835) bit-by-bit-identical: 15386 (-29) not-bit-by-bit-identical: 282 (+42) not-bit-by-bit-identicalcheck: 282 (+43) https://rb.zq1.de/compare.factory-20250131/graph.png shows the change over time https://rb.zq1.de/compare.factory-20250131/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 1 were in ring0 22 were in ring1 That makes it 23/4043 => 0.57 % which is below the overall average of 175/15668 => 1.12 % 282/15668 => 1.80 % of packages are not perfectly reproducible package notes: ==> ./FreeCAD/.rb.notes <== SR 1233273 fixes FTBFS but patch differs from URL ==> ./Komikku/.rb.notes <== => SR 1238506 Komikku nocheck [ 69s] Processing files: Komikku-lang-1.67.0-1.1.noarch ==> ./OpenImageIO/.rb.notes <== nachbau /usr/share/man/man1/maketx.1 differs (troff or preprocessor input, ASCII text, with very long lines) ==> ./abseil-cpp/.rb.notes <== => SR 1237987 1237960 = https://bugzilla.opensuse.org/show_bug.cgi?id=1235867 race? 1-core-VM still varies ; nothing helps ==> ./alex/.rb.notes <== build-id, ghc debuginfo="--debuginfo" rbk ==> ./build/.rb.notes <== nachbau: https://github.com/openSUSE/obs-build/pull/1054 ==> ./cargo-packaging/.rb.notes <== => SR 1233216 toolchain for rusty_v8 ==> ./chezmoi/.rb.notes <== FTBFS -j1 go1.19 parallelism ==> ./dealii/.rb.notes <== doxygen parallelism/race ==> ./deepin-gir-generator/.rb.notes <== debuginfo=--debuginfo multibuildrbkall ; nothing helps. /usr/share/go/1.17/contrib/src/pkg.deepin.io/gir/gudev-1.0/gudev.gen.c 1f61e81a27f2d77e4fcbc3f34de5f22c0be5960ba523fd1e9375a62257626e2b 0 ==> ./dunst/.rb.notes <== => https://github.com/dunst-project/dunst/pull/1435 date --- old /usr/bin/dunst (objdump) ==> ./dvgt/.rb.notes <== date .dvi toolchain from tex sysguide ==> ./eclipse-egit/.rb.notes <== => SR 1239889 strip-nondeterminism in /usr/share/eclipse/droplets/egit-egit/plugins/org.eclipse.egit.core.junit.source_5.11.0.202103091610-r.jar ==> ./erlang/.rb.notes <== FTBFS-j1 https://bugzilla.opensuse.org/show_bug.cgi?id=1120947 date+time ==> ./exaile/.rb.notes <== => https://github.com/exaile/exaile/pull/956 parallelism/race in exaile.desktop ==> ./examine/.rb.notes <== cosmic issue ==> ./forgejo-guardian/.rb.notes <== rust/llvm toolchain? nothing helps ==> ./form/.rb.notes <== missing doc files FTBFS-j1 : osc build --vm-type=kvm -j1 --noservice standard ==> ./gawk/.rb.notes <== => SR 1240443 PGO/bug ==> ./ghc-wai-app-static/.rb.notes <== ghc toolchain? debuginfo=--debuginfo ; nothing helps ; minor ==> ./ghc-yaml/.rb.notes <== ghc toolchain? debuginfo=--debuginfo ; nothing helps ==> ./ghc/.rb.notes <== ordering issue in /usr/lib64/ghc-9.8.3/lib/x86_64-linux-ghc-9.8.3/libHSCabal-3.10.3.0-8d80-ghc9.8.3.so ; nothing helps ==> ./gimp/.rb.notes <== = https://gitlab.gnome.org/GNOME/gimp-data/-/issues/7 gimp-splash.png date+time from gimp-data/images/meson.build command: [ gimp_exe ==> ./intel-graphics-compiler/.rb.notes <== = https://github.com/intel/intel-graphics-compiler/issues/359 ASLR build has variations in some (likely generated) source files: /home/abuild/rpmbuild/BUILD/build/IGC/VectorCompiler/lib/BiF/VCBuiltins64.cpp /home/abuild/rpmbuild/BUILD/build/IGC/Release/bif/embedder/OCLBiFImpl__igc_bif_BIFBC_122.cpp ==> ./ioquake3/.rb.notes <== => https://github.com/ioquake/ioq3/pull/704 debugsource: date+time ==> ./java-21-openjdk/.rb.notes <== race influence in libjvm.so debuginfo 2042=good ==> ./joker/.rb.notes <== => SR 1240852 1240514 #=> https://github.com/candid82/joker/commit/d2817bb18cd9cc0274460113a97dc87f2613d1ad #= https://github.com/candid82/joker/issues/491 hash random order ==> ./julia/.rb.notes <== stuck in -j1 build => workaround -j3 ==> ./kbd/.rb.notes <== 2024-12 https://github.com/legionus/kbd/commit/eebaa3b69efd9e3d218f3436dc43ff3340020ef5 ==> ./kernel-default-base/.rb.notes <== nachbau needs different calculation of release value release=31.51 nachbau worked ==> ./kicad/.rb.notes <== => SR 1238205 => https://gitlab.com/kicad/code/kicad/-/merge_requests/2087 tar from ==> ./libmfx-gen/.rb.notes <== vmtype=--vm-type=podman rbk uname -r? ==> ./libnice/.rb.notes <== FTBFS-j1 ==> ./librespeed-cli/.rb.notes <== #=> SR 1235815 1233732 date ==> ./libsoup/.rb.notes <== FTBFS-2040 found range good=2239732379 bad=2239734304 ==> ./liburing/.rb.notes <== FTBFS-j1? ==> ./llamacpp/.rb.notes <== #=> https://github.com/ggerganov/llama.cpp/pull/11366 CPU #= https://github.com/ggerganov/llama.cpp/issues/11306 random order ==> ./llvm16/.rb.notes <== = https://github.com/llvm/llvm-project/issues/72206 libomp LTO issue ==> ./llvm17/.rb.notes <== ASLR left in llvm17 .so /usr/lib64/libomp.so parallelism ==> ./lout/.rb.notes <== .pdf from ghostscript/gs, toolchain ==> ./mariadb/.rb.notes <== FTBFS-date found range good=1893453942 bad=1893455867 ==> ./mayavi/.rb.notes <== FTBFS-j1 / other? tvtk_classes.zip varies (mtimes?) ==> ./messagelib/.rb.notes <== parallelism in /usr/share/doc/qt6/KPim6MessageViewer.qch - toolchain? ==> ./mingw32-zlib/.rb.notes <== debuginfo=--debuginfo multibuildrbk ==> ./mousai/.rb.notes <== rust/llvm toolchain LTO? debuginfo=--debuginfo multibuildrbk ==> ./mvapich3/.rb.notes <== -%doc '$(/tmp/scr-FWqrmc)'./README.mvapich3-ucx_3_0-gnu-hpc-macros-devel +%doc '$(/tmp/scr-A6yaaM)'./README.mvapich3-ucx_3_0-gnu-hpc-macros-devel ==> ./nethack/.rb.notes <== => SR 1234650 parallelism/race /usr/lib/nethack/nethack ==> ./nodejs-electron/.rb.notes <== random build ID order issue ;huge pkg ==> ./nvidia-modprobe/.rb.notes <== => SR 1239771 date in /usr/share/man/man1/nvidia-modprobe.1.gz ==> ./ocaml-extlib/.rb.notes <== parallelism in debuginfo=--debuginfo multibuildrbkall ==> ./openttd/.rb.notes <== ASLR into build-id ==> ./palo/.rb.notes <== => patch sent per email to Helge Deller ; date in man + binary --- old /usr/sbin/palo (objdump) ==> ./paraview/.rb.notes <== uname -r Qt5 rcc/qrc timestamp ==> ./perl-libconfigfile/.rb.notes <== => SR 1236852 strange nachbau diff from likely race new//usr/lib/perl5/vendor_perl/5.40.0/ConfigFile.pm 2011-08-28 18:42:11.000000000 +0000 ==> ./pgvector/.rb.notes <== => SR 1240185 1240183 https://github.com/pgvector/pgvector/pull/764 = https://bugzilla.opensuse.org/show_bug.cgi?id=1236351 CPU -march=native ==> ./presage/.rb.notes <== => SR 1233892 = https://bugzilla.opensuse.org/show_bug.cgi?id=1234897 ==> ./procps/.rb.notes <== = https://gitlab.com/procps-ng/procps/-/issues/362 FTBFS-2038 uptime --raw test osc build --noservice --clean --vm-type=kvm "--build-opt=--vm-custom-opt=-rtc base=2038-01-20T00:00:00" standard ==> ./python-Django4/.rb.notes <== => SR 1240318 FTBFS-2038 ==> ./python-python-datamatrix/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1236437 FTBFS disk<RAM ==> ./python-rdflib/.rb.notes <== FTBFS-j1 :doc (pinged packager in Slack) ==> ./python-sherpa/.rb.notes <== FTBFS-j1 FTBFS-date?random? rbkt 801 works ==> ./python-tuf/.rb.notes <== FTBFS-2030-01-01 found range good=1893454059 bad=1893455984 ==> ./python-vcrpy/.rb.notes <== FTBFS-j1 osc build --vm-type=kvm --clean --noservice -j1 standard ==> ./python-webauthn/.rb.notes <== = https://github.com/duo-labs/py_webauthn/issues/243 FTBFS-2025-02-02 found range good=1738490879 bad=1738492804 ==> ./python-wxPython/.rb.notes <== FTBFS-j1 [ 4372s] _____________________________ xml_Tests.test_xml2 ______________________________ ==> ./qore-ssh2-module/.rb.notes <== --- binaries.nachbau/qore-ssh2-module.spec 2024-05-26 11:58:33.000000000 +0000 +++ binaries/qore-ssh2-module.spec 2024-05-26 11:58:33.000000000 +0000 ==> ./qt6-webengine/.rb.notes <== copyright year: /usr/src/debug/qtwebengine-everywhere-src-6.7.2/build/src/core/RelWithDebInfo/x86_64/gen/qtwebengine/common/extensions/api/webrtc_desktop_capture_private.h Copyright 2041 The Chromium Authors Qt toolchain issue ; hard to debug because of slow compile time ==> ./river/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo from CPU ==> ./scummvm/.rb.notes <== ASLR in debuginfo /usr/bin/scummvm ==> ./sequoia-sq/.rb.notes <== FTBFS-2026-01-31 FTBFS-2026-10-05 found range good=1769878065 bad=1769880423 ==> ./spack/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1235144 => https://github.com/spack/spack/pull/36064 parallelism ==> ./stacktile/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo ==> ./static-initrd/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1234709 date+host+CPU => SR 1232164 ==> ./swipl/.rb.notes <== ASLR? ==> ./uki/.rb.notes <== pesign ==> ./waylock/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo, CPU ==> ./wayprompt/.rb.notes <== = https://github.com/ziglang/zig/issues/22663 zig toolchain random path in debuginfo --- old /usr/lib/debug/usr/bin/wayprompt.debug (objdump) ==> ./weblate/.rb.notes <== FTBFS-2038 found range good=2147480401 bad=2147482326 => SR 1194546 = https://github.com/WeblateOrg/weblate/issues/8556 FTBFS-2038 toolchain mercurial ==> ./xdg-desktop-portal/.rb.notes <== => SR 1234111 xdg-desktop-portal Sphinx doctrees ==> ./yast2-firstboot/.rb.notes <== -TIFF image data, little-endian, direntries=13, width=25, height=24, bps=8, compression=LZW, PhotometricInterpretation=BlackIsZero +TIFF image data, little-endian, direntries=13, height=24, bps=8, compression=LZW, PhotometricInterpretation=BlackIsZero, width=25 ==> ./zig/.rb.notes <== Todo: ASLR in debuginfo => SR 1241564 = https://github.com/ziglang/zig/issues/22663 zig toolchain, CPU-dependent path in debuginfo ==> ./zls/.rb.notes <== = https://github.com/zigtools/zls/issues/2138 CPU / zig toolchain ; random path in debuginfo=--debuginfo multibuildrbkall -/usr/src/debug/zls-0.13.0/zig-cache/o/dd3075db6d19db75877b8d14c5f6fc11/version_data_0.13.0.zig 0 (none) 100644 root root 0 4294967295