Hi, This month, I created several bugreports and patches, while working on the NLnet-funded project about creating a 100% bit-reproducible OS (RBOS) [1]. I also worked towards the bot that checks reproducibility of SRs to Factory. The prototype of the OBS-project-setup is in https://build.opensuse.org/project/show/home:bmwiedemann:reproducible:rebuild it already shows identical results with for r in j1 future1y openSUSE_Factory ; do osc api "/build/home:bmwiedemann:reproducible:rebuild/$r/x86_64/_repository?view=binaryversions" | grep libzzip ; done And different results with grep theunreproduciblepackage\\.rpm Additionally my reproducible-faketools got an overhaul to get rid of package-conflicts to make them even more versatile and useful, especially within OBS. They are now in Factory to allow easy testing in more places. Finally, I am happy to report that a bit-reproducible libreoffice is already in the devel-project. The Plan for RBOS: to reach the goal of a distribution that consists of 100% bit-reproducible packages, we need to address multiple issues. 1. newly unreproducible packages get submitted regularly and we need to test + notify these to slow down the introduction of new issues. A PoC works in https://build.opensuse.org/project/show/home:bmwiedemann:reproducible:rebuild by using a constant Release and distribution value, omitting disturl and adding in some variations via reproducible-faketools in the prjconf. TODO: This needs to be integrated in the workflows for staging + review of submissions to openSUSE:Factory. 2. Some (undroppable) existing packages have variations - several fixes were already submitted and more are to come. 3. Some packages built in OBS are modified by pesign to make them work with secure boot, e.g. grub2 and systemd - TODO - disable obs-pesign-integration? Build these outside of OBS as reference? Use a patched obs-pesign-integration version? 4. https://build.opensuse.org/project/show/home:bmwiedemann:reproducible:distribution:ring0 is already setup and ring1 will come soon with more packages required to build a MinimalVM image or maybe even a DVD. The only open issue in ring0 is with pam:full pdf timestamps from xmlgraphics-fop. [1] https://nlnet.nl/project/Reproducible-openSUSE/ Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/CDIFFRNRSCI5BBXW7QSQSZVKE45YFTTH/ Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2024-06/ I uploaded https://rb.zq1.de/compare.factory-20240731/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15666 (+117) build-tried: 15602 (+111) build-failed: 13 (-9) build-n-a: 186 (-7) build-succeeded: 15403 (+127) build-official-failed+na: 25 (-14) build-compare-failed: 154 (+0) build-compare-succeeded: 15249 (+127) verify-failed: 249 (+15) verified-semi-reproducible: 14944 (+89) verified-bit-identical: 13591 (+46) bit-by-bit-identical: 14779 (+157) not-bit-by-bit-identical: 625 (-22) not-bit-by-bit-identicalcheck: 624 (-30) https://rb.zq1.de/compare.factory-20240731/graph.png shows the change over time https://rb.zq1.de/compare.factory-20240731/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 3 were in ring0 31 were in ring1 That makes it 34/4043 => 0.84 % which is below the overall average of 154/15403 => 1.00 % 625/15403 => 4.06 % of packages are not perfectly reproducible package notes: ==> ./TeXmacs/.rb.notes <== => https://github.com/texmacs/texmacs/pull/77 date+time from `date` --- old /usr/libexec/TeXmacs/bin/texmacs.bin (objdump) ==> ./apache-arrow/.rb.notes <== OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --noservice --vm-type=kvm --clean standard ==> ./armagetron/.rb.notes <== => SR 1188202 #=> https://gitlab.com/armagetronad/armagetronad/-/merge_requests/162 date ==> ./bcc/.rb.notes <== TODO - redo in luajit src/lj_asm_x86.h asm_href ; also for neovim = https://github.com/moonjit/moonjit/issues/110 report CPU-detection in moonjit ==> ./certgen/.rb.notes <== Go buildinf: - go1.23rc1 ==> ./clamav/.rb.notes <== => SR 1190176 = https://github.com/Cisco-Talos/clamav/issues/1300 FTBFS-2024-07-28 found range good=1722124782 bad=1722126706 ==> ./cloudflared/.rb.notes <== => SR 1188166 => https://github.com/cloudflare/cloudflared/pull/1289 date from `date` --- old /usr/bin/cloudflared (objdump) ==> ./dpdk/.rb.notes <== => SR 1185443 Sphinx doctrees regression ==> ./emacs/.rb.notes <== .pdmp from ["./temacs" "--__aslr-disabled" "-batch" "--no-build-details" "-l" "loadup" "--temacs=pdump" "--bin-dest" "/usr/bin/" "--eln-dest" "/usr/lib64/emacs/29.4/"] ==> ./fonttosfnt/.rb.notes <== => SR 1190278 => https://gitlab.freedesktop.org/xorg/app/fonttosfnt/-/merge_requests/22 toolchain, date ==> ./fractal/.rb.notes <== rust/llvm symbol order variation ==> ./gegl/.rb.notes <== => SR 1188550 = https://gitlab.gnome.org/GNOME/gegl/-/issues/337 parallelism, memory +++ new//usr/share/gir-1.0/Gegl-0.4.gir 2023-06-26 00:00:00.000000000 +0000 ==> ./gettext-runtime/.rb.notes <== => SR 1188059 1187694 jar mtime in /usr/lib64/gettext/gettext.jar = https://lists.gnu.org/archive/html/bug-gettext/2024-07/msg00020.html ==> ./gnutls/.rb.notes <== date in man - probably because patch updates mtime # filterdiff zcat R*/usr/share/man/man1/gnutls-cli.1.gz ==> ./gromacs/.rb.notes <== FTBFS-j1 failed - needs 4+ cores succeeds with parallelism=6 parallelism2=5 multibuildrbkall ==> ./helm/.rb.notes <== FTBFS-2028-04-05 2032-08-21 SSL ==> ./java-jwt/.rb.notes <== java/javadoc filesys (low-entropy) + strip-nd + other jar from ["/usr/lib64/jvm/java-21-openjdk-21/bin/java", "-classpath", "/usr/share/java/xmvn/xmvn-install.jar:/usr/share/java/xmvn/xmvn-api.jar:/usr/share/java/xmvn/xmvn-core.jar:/usr/share/java/beust-jcommander.jar:/usr/share/java/slf4j/api.jar:/usr/share/java/slf4j/simple.jar:/usr/share/java/objectweb-asm/asm.jar:/usr/share/java/commons-compress.jar:/usr/share/java/commons-io.jar", "org.fedoraproject.xmvn.tools.install.cli.InstallerCli", "-R", ".xmvn-reactor", "-n", "java-jwt", "-d", "/home/abuild/rpmbuild/BUILDROOT/java-jwt-3.8.3-0.x86_64"] ==> ./jaxen/.rb.notes <== post-processing modifies mtime in .jar /usr/bin/python3 /usr/share/java-utils/maven_depmap.py --pom-base /home/abuild/rpmbuild/BUILDROOT/jaxen-2.0.0-1.1.x86_64/usr/share/maven-poms --jar-base /home/abuild/rpmbuild/BUILDROOT/jaxen-2.0.0-1.1.x86_64/usr/share/java /home/abuild/rpmbuild/BUILDROOT/jaxen-2.0.0-1.1.x86_64/usr/share/maven-metadata/jaxen.xml /home/abuild/rpmbuild/BUILDROOT/jaxen-2.0.0-1.1.x86_64/usr/share/maven-poms/jaxen.pom /home/abuild/rpmbuild/BUILDROOT/jaxen-2.0.0-1.1.x86_64/usr/share/java/jaxen.jar ==> ./jmock/.rb.notes <== TODO: .jar javadoc jar from ["/usr/lib64/jvm/java-21-openjdk-21/bin/java", "-Dant.tstamp.now=1708473600", "-classpath", "/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/share/java/ant/ant-junit.jar:/usr/share/java/junit.jar:/home/abuild/rpmbuild/BUILD/jmock-1.2.0/build/classes:/usr/share/java/cglib/cglib.jar:/usr/share/java/cglib/cglib-sample.jar", "-Dant.home=/usr/share/ant", "-Dant.library.dir=/usr/share/ant/lib", "org.apache.tools.ant.launch.Launcher", "-cp", "", "-Dant.build.javac.source=1.8", "-Dant.build.javac.target=1.8", "-Dbuild.sysclasspath=only", "package"] ==> ./kernel-source/.rb.notes <== doc variation from Sphinx -j auto ==> ./kf6-kirigami/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228131 parallelism via /home/abuild/rpmbuild/BUILD/kirigami-6.4.0/build/src/dialogs/.rcc/qmlcache/KirigamiDialogs_PromptDialog_qml.cpp ==> ./kf6-qqc2-desktop-style/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228131 qt6-declarative toolchain issue? parallelism? +++ /var/tmp/build-root.2b/.mount/home/abuild/rpmbuild/BUILD/qqc2-desktop-style-6.0.0/build/.rcc/qmlcache/org_kde_desktop_private_org.kde.desktop/private/CheckIndicator_qml.cpp 2040-04-14 03:00:38.066666668 +0000 ==> ./kubernetes1.26/.rb.notes <== => SR 1190449 random go tmp build path ==> ./lapackpp/.rb.notes <== => https://github.com/icl-utk-edu/lapackpp/pull/68 hostname in /usr/include/lapack/defines.h from cmake + defines.h.in ==> ./latex2html/.rb.notes <== => SR 1188512 nochecks cause variation => SR 1150775 drop latex log ==> ./ldns/.rb.notes <== [ 45s] ./libtool --tag=CC --quiet --mode=link gcc -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -fno-strict-aliasing -flto=auto examples/ldns-nsec3-hash.lo compat/b64_pton.lo compat/b64_ntop.lo libldns.la -lssl -lcrypto -o examples/ldns-nsec3-hash^M ==> ./libdb-4_8/.rb.notes <== => SR 1190247 1187675 /usr/share/java/db-4.8.30.jar mtimes from by pid=171773 dir=/home/abuild/rpmbuild/BUILD/db-4.8.30/build_nptl/classes exec="/usr/bin/jar", ["jar", "cfm", "../db.jar", "../../dist/../java/jarManifestEntries", "./com/sleepycat"] - started ==> ./libguestfs/.rb.notes <== minor /usr/share/doc/packages/rubygem-libguestfs/api/created.rid ==> ./librcc/.rb.notes <== => SR 1188204 #=> https://github.com/RusXMMS/librcc/pull/5 date --- old//usr/share/doc/packages/librcc-devel/examples/Makefile 2022-12-09 00:00:00.000000000 +0000 ==> ./libreoffice/.rb.notes <== => SR 1189287 = https://gerrit.libreoffice.org/q/topic:reprobuild clucene, .jar mtime, .zip mtime ==> ./libzypp/.rb.notes <== #= https://github.com/openSUSE/libzypp/issues/559 FTBFS-2038 stuck, bug osc build --noservice --vm-type=kvm --build-opt=--vm-custom-opt="-rtc base=2040-01-25T00:00:00" standard ==> ./llvm17/.rb.notes <== = https://github.com/llvm/llvm-project/issues/72206 Marvin investigates. Partially from ASLR - uninitialized memory? ==> ./maliit-keyboard/.rb.notes <== => SR 1185254 maliit-keyboard nocheck ==> ./mozilla-nss/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1081723 => https://bugzilla.mozilla.org/show_bug.cgi?id=1902078 = https://bugzilla.mozilla.org/show_bug.cgi?id=1813401 FTBFS-2023 ==> ./nautilus/.rb.notes <== => https://gitlab.gnome.org/GNOME/nautilus/-/merge_requests/1555 date from data/org.gnome.Nautilus.metainfo.xml.in.in: <release version="@release-version@" date="@build-date@"/> ==> ./neovim/.rb.notes <== = https://github.com/neovim/neovim/issues/26387 - maybe toolchain https://github.com/LuaJIT/LuaJIT/issues/1008 ==> ./newtonsoft-json/.rb.notes <== mono, random 20 byte ==> ./nodejs22/.rb.notes <== FTBFS-SSL: test/fixtures/x509-escaping/*pem expires 2031-12-18 ==> ./openblas/.rb.notes <== => SR 1190320 = https://bugzilla.opensuse.org/show_bug.cgi?id=1228177 CPU = https://bugzilla.opensuse.org/show_bug.cgi?id=1181083 FTBFS-j1 ==> ./openssl-1_0_0/.rb.notes <== verification issue ==> ./openssl-3/.rb.notes <== => SR 1187438 = https://bugzilla.opensuse.org/show_bug.cgi?id=1223336 random debugsource FTBFS-2035-07-02 ==> ./pop-launcher/.rb.notes <== parallelism ==> ./python-Django4/.rb.notes <== FTBFS-2038 ==> ./python-Sphinx/.rb.notes <== => https://github.com/sphinx-doc/sphinx/pull/12606 gzip mtime 0 0 0 0 0 1 ==> ./python-cfn-lint/.rb.notes <== FTBFS-2024-10-13 OSC_BUILD_ROOT=/var/tmp/build-root.$slot time osc build --noservice --vm-type=kvm --build-opt=--vm-custom-opt="-rtc base=2024-10-14T00:00:00" standard ==> ./python-contourpy/.rb.notes <== random: toolchain meson-py? ==> ./python-libcst/.rb.notes <== unknown rust/llvm /usr/lib64/python3.11/site-packages/libcst/native.cpython-311-x86_64-linux-gnu.so ==> ./python-paho-mqtt/.rb.notes <== FTBFS-j1 + SSL expired =https://github.com/eclipse/paho.mqtt.python/pull/854 found range 1783313692 -> 1783315617 ==> ./python-paramiko/.rb.notes <== FTBFS-CPU+j1 ==> ./python-pygraphviz/.rb.notes <== captures execution time ==> ./python-pysnmp/.rb.notes <== => https://github.com/lextudio/pysnmp/pull/35 FTBFS-2038 ==> ./python-pytest-mpi/.rb.notes <== FTBFS-j1 + CPU osc build --clean --vm-type=kvm -j1 --noservice --clean standard ==> ./python-python-datamatrix/.rb.notes <== FTBFS-j1 OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --vm-type=kvm --clean -j1 --noservice standard ==> ./python-ruff/.rb.notes <== ASLR+checks/PID?, low-entropy? causes binary diff, size diff ==> ./python-spyder-notebook/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228441 report FTBFS OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --noservice --vm-type=kvm --clean -j4 standard ==> ./python310/.rb.notes <== PGO + other(:doc) ==> ./python311/.rb.notes <== => https://github.com/sphinx-doc/sphinxcontrib-devhelp/pull/13 gzip mtime => https://github.com/python/cpython/pull/121872 python311:doc date, partial ==> ./python313/.rb.notes <== = https://github.com/python/cpython/issues/122433 FTBFS-j1 OSC_BUILD_ROOT=/var/tmp/build-root.$slot time osc build --noservice --vm-type=kvm --clean standard ==> ./rabbitmq-server/.rb.notes <== FTBFS https://build.opensuse.org/package/show/network:messaging:amqp/rabbitmq-server ==> ./rmt-server/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1227542 date -/usr/lib64/rmt/vendor/bundle/ruby/3.3.0/gems/base32-0.3.4/lib/base32 0 (none) 40777 root root 0 4294967295 ==> ./samba/.rb.notes <== #= https://bugzilla.opensuse.org/show_bug.cgi?id=1225754 parallelism+ASLR --- /home/abuild/rpmbuild/BUILD/samba-4.20.1+git.335.0a46cdafe2/bin/default/source3/librpc/gen_ndr/py_smbXsrv.c 2024-05-31 03:19:30.679999998 +0000 ==> ./systemd/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228091 pesign = https://bugzilla.opensuse.org/show_bug.cgi?id=1226200 FTBFS-2038 ==> ./tigervnc/.rb.notes <== minor jar mtimes ; from jar call #= https://bugzilla.opensuse.org/show_bug.cgi?id=1208478 RSA key ==> ./turbo/.rb.notes <== minor issue in /usr/lib64/libturbo-core.so ; ASLR Binary files /var/tmp/build-root.20/.mount/home/abuild/rpmbuild/BUILD/turbo-1715766145.697580e/build/CMakeFiles/turbo-core.dir/Unity/unity_0_cxx.cxx.o and /var/tmp/build-root.20b/.mount/home/abuild/rpmbuild/BUILD/turbo-1715766145.697580e/build/CMakeFiles/turbo-core.dir/Unity/unity_0_cxx.cxx.o differ ==> ./warzone2100/.rb.notes <== = https://github.com/BinomialLLC/basis_universal/issues/374 report parallelism = https://github.com/Warzone2100/warzone2100/issues/2991 report parallelism-dependent output ==> ./whatsie/.rb.notes <== date needs https://codereview.qt-project.org/gitweb?p=qt%2Fqtbase.git;a=commit;h=297fe90329782d35dc4f155185ded045551ef52f backported into libqt5-qtbase ==> ./wireplumber/.rb.notes <== parallelism: searchindex.js varies, toolchain python-Sphinx from ["/usr/bin/sphinx-build", "-q", "-E", "-j", "auto", "-d", "docs/html.p", "-c", "docs", "../docs/rst", "docs/html"] ==> ./xmobar/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228175 ==> ./xorg-x11-fonts/.rb.notes <== => https://gitlab.freedesktop.org/xorg/app/fonttosfnt/-/merge_requests/22 toolchain fonttosfnt, -M=converted, ASLR, timestamp? = https://bugzilla.opensuse.org/show_bug.cgi?id=1173396 ASLR : x*3 uninit bytes 'checksum adjustment' ==> ./zola/.rb.notes <== unknown rust/llvm