[2025-11-13T17:11:54.448Z] Triggered by CI message. [2025-11-13T17:11:54.449Z] Checking out git https://github.com/coreos/coreos-ci into /var/lib/jenkins/jobs/bodhi-trigger/workspace@script/1e4fe20c43e6e811f5c65301638e3ca356eacc690aa103cf9763ce056ed9ad26 to read jobs/bodhi-trigger.Jenkinsfile [2025-11-13T17:11:54.450Z] The recommended git tool is: NONE [2025-11-13T17:11:54.450Z] No credentials specified [2025-11-13T17:11:54.450Z] > git rev-parse --resolve-git-dir /var/lib/jenkins/jobs/bodhi-trigger/workspace@script/1e4fe20c43e6e811f5c65301638e3ca356eacc690aa103cf9763ce056ed9ad26/.git # timeout=10 [2025-11-13T17:11:54.457Z] Fetching changes from the remote Git repository [2025-11-13T17:11:54.458Z] > git config remote.origin.url https://github.com/coreos/coreos-ci # timeout=10 [2025-11-13T17:11:54.465Z] Fetching upstream changes from https://github.com/coreos/coreos-ci [2025-11-13T17:11:54.465Z] > git --version # timeout=10 [2025-11-13T17:11:54.473Z] > git --version # 'git version 2.47.1' [2025-11-13T17:11:54.473Z] > git fetch --tags --force --progress -- https://github.com/coreos/coreos-ci +refs/heads/*:refs/remotes/origin/* # timeout=10 [2025-11-13T17:11:54.602Z] > git rev-parse origin/main^{commit} # timeout=10 [2025-11-13T17:11:54.610Z] Checking out Revision 73431a218c32c83c8d5378c364ac11b2391f7be5 (origin/main) [2025-11-13T17:11:54.610Z] > git config core.sparsecheckout # timeout=10 [2025-11-13T17:11:54.616Z] > git checkout -f 73431a218c32c83c8d5378c364ac11b2391f7be5 # timeout=10 [2025-11-13T17:11:54.626Z] Commit message: "test-override: always buildfetch" [2025-11-13T17:11:54.626Z] > git rev-list --no-walk 73431a218c32c83c8d5378c364ac11b2391f7be5 # timeout=10 [2025-11-13T17:11:54.662Z] Loading library coreos@main [2025-11-13T17:11:54.662Z] Attempting to resolve main from remote references... [2025-11-13T17:11:54.662Z] > git --version # timeout=10 [2025-11-13T17:11:54.668Z] > git --version # 'git version 2.47.1' [2025-11-13T17:11:54.668Z] > git ls-remote -- https://github.com/coreos/coreos-ci-lib.git # timeout=10 [2025-11-13T17:11:54.790Z] Found match: refs/heads/main revision 6a64d66a9bc5979e0ef3bcdd7ad1a05813b7a671 [2025-11-13T17:11:54.794Z] The recommended git tool is: NONE [2025-11-13T17:11:54.794Z] No credentials specified [2025-11-13T17:11:54.795Z] > git rev-parse --resolve-git-dir /var/lib/jenkins/jobs/bodhi-trigger/workspace@libs/ff60277bd48d76a282f8da417470514d8522d9eac9fc125da5f550649b50d86b/.git # timeout=10 [2025-11-13T17:11:54.801Z] Fetching changes from the remote Git repository [2025-11-13T17:11:54.802Z] > git config remote.origin.url https://github.com/coreos/coreos-ci-lib.git # timeout=10 [2025-11-13T17:11:54.809Z] Fetching without tags [2025-11-13T17:11:54.809Z] Fetching upstream changes from https://github.com/coreos/coreos-ci-lib.git [2025-11-13T17:11:54.809Z] > git --version # timeout=10 [2025-11-13T17:11:54.816Z] > git --version # 'git version 2.47.1' [2025-11-13T17:11:54.816Z] > git fetch --no-tags --force --progress -- https://github.com/coreos/coreos-ci-lib.git +refs/heads/*:refs/remotes/origin/* # timeout=10 [2025-11-13T17:11:54.928Z] Checking out Revision 6a64d66a9bc5979e0ef3bcdd7ad1a05813b7a671 (main) [2025-11-13T17:11:54.928Z] > git config core.sparsecheckout # timeout=10 [2025-11-13T17:11:54.934Z] > git checkout -f 6a64d66a9bc5979e0ef3bcdd7ad1a05813b7a671 # timeout=10 [2025-11-13T17:11:54.943Z] Commit message: "cosaBuild: call `cosa osbuild` separately" [2025-11-13T17:11:54.943Z] > git rev-list --no-walk 6a64d66a9bc5979e0ef3bcdd7ad1a05813b7a671 # timeout=10 [2025-11-13T17:11:55.029Z] [Pipeline] Start of Pipeline [2025-11-13T17:11:55.056Z] [Pipeline] node [2025-11-13T17:11:55.066Z] Running on Jenkins in /var/lib/jenkins/jobs/bodhi-trigger/workspace [2025-11-13T17:11:55.079Z] [Pipeline] { [2025-11-13T17:11:55.104Z] [Pipeline] checkout [2025-11-13T17:11:55.113Z] The recommended git tool is: NONE [2025-11-13T17:11:55.113Z] No credentials specified [2025-11-13T17:11:55.114Z] > git rev-parse --resolve-git-dir /var/lib/jenkins/jobs/bodhi-trigger/workspace/.git # timeout=10 [2025-11-13T17:11:55.121Z] Fetching changes from the remote Git repository [2025-11-13T17:11:55.124Z] > git config remote.origin.url https://github.com/coreos/coreos-ci # timeout=10 [2025-11-13T17:11:55.131Z] Fetching upstream changes from https://github.com/coreos/coreos-ci [2025-11-13T17:11:55.131Z] > git --version # timeout=10 [2025-11-13T17:11:55.136Z] > git --version # 'git version 2.47.1' [2025-11-13T17:11:55.136Z] > git fetch --tags --force --progress -- https://github.com/coreos/coreos-ci +refs/heads/*:refs/remotes/origin/* # timeout=10 [2025-11-13T17:11:55.249Z] > git rev-parse origin/main^{commit} # timeout=10 [2025-11-13T17:11:55.256Z] Checking out Revision 73431a218c32c83c8d5378c364ac11b2391f7be5 (origin/main) [2025-11-13T17:11:55.256Z] > git config core.sparsecheckout # timeout=10 [2025-11-13T17:11:55.265Z] > git checkout -f 73431a218c32c83c8d5378c364ac11b2391f7be5 # timeout=10 [2025-11-13T17:11:55.274Z] Commit message: "test-override: always buildfetch" [2025-11-13T17:11:55.291Z] [Pipeline] readYaml [2025-11-13T17:11:55.370Z] [Pipeline] withEnv [2025-11-13T17:11:55.374Z] [Pipeline] { [2025-11-13T17:11:55.398Z] [Pipeline] sh [2025-11-13T17:11:55.659Z] + set -xeuo pipefail [2025-11-13T17:11:55.660Z] ++ umask [2025-11-13T17:11:55.660Z] + '[' 0022 = 0000 ']' [2025-11-13T17:11:55.660Z] + rm -rf pipe [2025-11-13T17:11:55.660Z] + git clone https://github.com/coreos/fedora-coreos-pipeline --depth=1 pipe [2025-11-13T17:11:55.660Z] Cloning into 'pipe'... [2025-11-13T17:11:55.674Z] [Pipeline] } [2025-11-13T17:11:55.697Z] [Pipeline] // withEnv [2025-11-13T17:11:55.713Z] [Pipeline] load [2025-11-13T17:11:55.821Z] [Pipeline] { (pipe/utils.groovy) [2025-11-13T17:11:55.839Z] Did you forget the `def` keyword? Script1 seems to be setting a field named PROTECTED_JENKINSES (to a value of type LinkedHashMap) which could lead to memory leaks or other issues. [2025-11-13T17:11:55.843Z] [Pipeline] } [2025-11-13T17:11:55.866Z] [Pipeline] // load [2025-11-13T17:11:55.871Z] Did you forget the `def` keyword? WorkflowScript seems to be setting a field named pipeutils (to a value of type Script1) which could lead to memory leaks or other issues. [2025-11-13T17:11:55.880Z] [Pipeline] readYaml [2025-11-13T17:11:55.900Z] Did you forget the `def` keyword? WorkflowScript seems to be setting a field named pipecfg (to a value of type LinkedHashMap) which could lead to memory leaks or other issues. [2025-11-13T17:11:55.903Z] Did you forget the `def` keyword? WorkflowScript seems to be setting a field named streams (to a value of type ArrayList) which could lead to memory leaks or other issues. [2025-11-13T17:11:55.903Z] Did you forget the `def` keyword? WorkflowScript seems to be setting a field named stream_to_releasever (to a value of type LinkedHashMap) which could lead to memory leaks or other issues. [2025-11-13T17:11:55.917Z] [Pipeline] withEnv [2025-11-13T17:11:55.921Z] [Pipeline] { [2025-11-13T17:11:55.944Z] [Pipeline] sh [2025-11-13T17:11:56.205Z] + set -xeuo pipefail [2025-11-13T17:11:56.205Z] ++ umask [2025-11-13T17:11:56.205Z] + '[' 0022 = 0000 ']' [2025-11-13T17:11:56.205Z] + curl -sSLO https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/manifest.yaml [2025-11-13T17:11:56.219Z] [Pipeline] } [2025-11-13T17:11:56.246Z] [Pipeline] // withEnv [2025-11-13T17:11:56.259Z] [Pipeline] readYaml [2025-11-13T17:11:56.291Z] [Pipeline] withEnv [2025-11-13T17:11:56.295Z] [Pipeline] { [2025-11-13T17:11:56.314Z] [Pipeline] sh [2025-11-13T17:11:56.575Z] + set -xeuo pipefail [2025-11-13T17:11:56.575Z] ++ umask [2025-11-13T17:11:56.575Z] + '[' 0022 = 0000 ']' [2025-11-13T17:11:56.575Z] + curl -sSLO https://raw.githubusercontent.com/coreos/fedora-coreos-config/rawhide/manifest.yaml [2025-11-13T17:11:56.591Z] [Pipeline] } [2025-11-13T17:11:56.618Z] [Pipeline] // withEnv [2025-11-13T17:11:56.633Z] [Pipeline] readYaml [2025-11-13T17:11:56.650Z] Did you forget the `def` keyword? WorkflowScript seems to be setting a field named releasevers (to a value of type LinkedHashSet) which could lead to memory leaks or other issues. [2025-11-13T17:11:56.653Z] [Pipeline] } [2025-11-13T17:11:56.685Z] [Pipeline] // node [2025-11-13T17:11:56.702Z] [Pipeline] properties [2025-11-13T17:11:56.724Z] [Pipeline] echo [2025-11-13T17:11:56.729Z] Handling message: {"artifact":{"type":"koji-build-group","builds":[{"type":"koji-build","id":2854791,"task_id":138735530,"nvr":"android-file-transfer-4.5-4.fc44"},{"type":"koji-build","id":2858131,"task_id":138828734,"nvr":"apfs-fuse-0-31.20200928gitee71aa5.fc44"},{"type":"koji-build","id":2854792,"task_id":138735528,"nvr":"apptainer-1.4.4-2.fc44"},{"type":"koji-build","id":2858133,"task_id":138828736,"nvr":"bindfs-1.18.3-2.fc44"},{"type":"koji-build","id":2854794,"task_id":138735529,"nvr":"btrfs-fuse-0-19.20230119git812c4b7.fc44"},{"type":"koji-build","id":2854793,"task_id":138739082,"nvr":"buildbox-1.3.42-2.fc44"},{"type":"koji-build","id":2854795,"task_id":138735526,"nvr":"ceph-20.1.1-5.fc44"},{"type":"koji-build","id":2854790,"task_id":138735525,"nvr":"charliecloud-0.42-1.fc44"},{"type":"koji-build","id":2854799,"task_id":138735652,"nvr":"e2fsprogs-1.47.3-3.fc44"},{"type":"koji-build","id":2854796,"task_id":138735543,"nvr":"erofs-utils-1.8.10-3.fc44"},{"type":"koji-build","id":2854890,"task_id":138741442,"nvr":"freerdp-3.17.2-3.fc44"},{"type":"koji-build","id":2854789,"task_id":138735199,"nvr":"fuse3-3.17.4-1.fc44"},{"type":"koji-build","id":2854855,"task_id":138739129,"nvr":"fuse-btfs-3.1-3.fc44"},{"type":"koji-build","id":2854857,"task_id":138739480,"nvr":"fuse-overlayfs-1.13-5.fc44"},{"type":"koji-build","id":2854856,"task_id":138739244,"nvr":"fuse-sshfs-3.7.3-13.fc44"},{"type":"koji-build","id":2854865,"task_id":138739922,"nvr":"gnome-connections-49.0-2.fc44"},{"type":"koji-build","id":2854875,"task_id":138742374,"nvr":"gnome-remote-desktop-49.1-2.fc44"},{"type":"koji-build","id":2854892,"task_id":138741765,"nvr":"grub2-2.12-46.fc44"},{"type":"koji-build","id":2858841,"task_id":138860704,"nvr":"gvfs-1.58.0-4.fc44"},{"type":"koji-build","id":2854895,"task_id":138741762,"nvr":"kio-fuse-5.1.1-2.fc44"},{"type":"koji-build","id":2854893,"task_id":138745487,"nvr":"libnbd-1.23.10-2.fc44"},{"type":"koji-build","id":2854896,"task_id":138745488,"nvr":"libvirt-11.9.0-2.fc44"},{"type":"koji-build","id":2854898,"task_id":138741847,"nvr":"open-vm-tools-13.0.0-2.fc44"},{"type":"koji-build","id":2854899,"task_id":138742007,"nvr":"ostree-2025.6-3.fc44"},{"type":"koji-build","id":2854910,"task_id":138742543,"nvr":"partclone-0.3.38-2.fc44"},{"type":"koji-build","id":2854904,"task_id":138742073,"nvr":"qemu-10.1.2-4.fc44"},{"type":"koji-build","id":2854912,"task_id":138742553,"nvr":"wimlib-1.14.4-6.fc44"},{"type":"koji-build","id":2854915,"task_id":138742573,"nvr":"xdg-desktop-portal-1.20.3-4.fc44"},{"type":"koji-build","id":2854916,"task_id":138742580,"nvr":"xmount-1.2.0-5.fc44"},{"type":"koji-build","id":2854921,"task_id":138742647,"nvr":"xrdp-0.10.4-4.fc44"},{"type":"koji-build","id":2860001,"task_id":138886685,"nvr":"flatpak-1.16.1-13.fc44"}]},"agent":"spot","re-trigger":false,"update":{"autokarma":true,"autotime":true,"stable_karma":3,"stable_days":0,"unstable_karma":-3,"require_bugs":true,"require_testcases":true,"display_name":"","notes":"Update to fuse3 3.17.4. Fuse 3.17.1 bumped the SOVER, so we needed a side-tag to rebuild everything. The good news is: everything rebuilt with minimal change. Except for charliecloud, which... wasn't building at all? But there weren't bugs? I dunno, I fixed it. Oh, and there was a bug in freerdp's pkgconfig files which was already fixed upstream (and was keeping other packages from building), so I brought that patch in. Any way, everything should just keep working as it did before.\n\nFeel free to shower me with love or whiskey. Or do something nice for a stranger.\n\nP.S. I'm not planning on doing this for Fedora 43. Too much churn.\n\n\n\n","type":"bugfix","status":"testing","request":null,"severity":"unspecified","suggest":"unspecified","locked":false,"pushed":true,"critpath":true,"critpath_groups":"core critical-path-anaconda critical-path-apps critical-path-base critical-path-build critical-path-cloud critical-path-compose critical-path-coreos critical-path-gnome critical-path-kde critical-path-lxde critical-path-lxqt critical-path-server critical-path-xfce","close_bugs":true,"date_submitted":"2025-11-05 00:31:11","date_modified":"2025-11-13 17:11:50","date_approved":null,"date_testing":null,"date_stable":null,"alias":"FEDORA-2025-fb66eccf2c","test_gating_status":"failed","from_tag":"f44-build-side-122092","date_pushed":null,"meets_testing_requirements":false,"url":"https://bodhi.fedoraproject.org/updates/FEDORA-2025-fb66eccf2c","title":"android-file-transfer-4.5-4.fc44 apfs-fuse-0-31.20200928gitee71aa5.fc44 apptainer-1.4.4-2.fc44 bindfs-1.18.3-2.fc44 btrfs-fuse-0-19.20230119git812c4b7.fc44 buildbox-1.3.42-2.fc44 ceph-20.1.1-5.fc44 charliecloud-0.42-1.fc44 e2fsprogs-1.47.3-3.fc44 erofs-utils-1.8.10-3.fc44 flatpak-1.16.1-13.fc44 freerdp-3.17.2-3.fc44 fuse-btfs-3.1-3.fc44 fuse-overlayfs-1.13-5.fc44 fuse-sshfs-3.7.3-13.fc44 fuse3-3.17.4-1.fc44 gnome-connections-49.0-2.fc44 gnome-remote-desktop-49.1-2.fc44 grub2-2.12-46.fc44 gvfs-1.58.0-4.fc44 kio-fuse-5.1.1-2.fc44 libnbd-1.23.10-2.fc44 libvirt-11.9.0-2.fc44 open-vm-tools-13.0.0-2.fc44 ostree-2025.6-3.fc44 partclone-0.3.38-2.fc44 qemu-10.1.2-4.fc44 wimlib-1.14.4-6.fc44 xdg-desktop-portal-1.20.3-4.fc44 xmount-1.2.0-5.fc44 xrdp-0.10.4-4.fc44","version_hash":"1886e2b91ff8d993d6550d9f80f79d2f5937c4aa","release":{"name":"F44","long_name":"Fedora 44","version":"44","id_prefix":"FEDORA","branch":"rawhide","dist_tag":"f44","stable_tag":"f44","testing_tag":"f44-updates-testing","candidate_tag":"f44-updates-candidate","pending_signing_tag":"f44-signing-pending","pending_testing_tag":"f44-updates-testing-pending","pending_stable_tag":"f44-updates-pending","override_tag":"f44-override","mail_template":"fedora_errata_template","state":"pending","composed_by_bodhi":false,"create_automatic_updates":true,"package_manager":"unspecified","testing_repository":null,"released_on":null,"eol":null,"critpath_mandatory_days_in_testing":0,"mandatory_days_in_testing":0,"critpath_min_karma":0,"min_karma":0,"setting_status":"pre_beta"},"user":{"name":"spot","email":"spotrh@gmail.com","id":224,"avatar":null,"openid":null,"groups":[{"name":"provenpackager"},{"name":"packager"},{"name":"arm-qa"},{"name":"cvsfedora"},{"name":"mirror-ia64"},{"name":"gitfedora-logos"},{"name":"fedorabugs"},{"name":"ambassadors"},{"name":"signed_fpca"},{"name":"openhw2012"},{"name":"paste-deleter"},{"name":"3d-printing-sig"},{"name":"cla_redhat"},{"name":"fedora-contributor"},{"name":"gitgeneric-release"},{"name":"gitmoksha"},{"name":"irc-support-operators"},{"name":"ipausers"},{"name":"r-maint-sig"}]},"comments":[{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'waiting'.","timestamp":"2025-11-05 00:31:11","update_id":778301,"user_id":91,"id":4414824,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'failed'.","timestamp":"2025-11-05 00:46:33","update_id":778301,"user_id":91,"id":4414828,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'waiting'.","timestamp":"2025-11-05 01:49:35","update_id":778301,"user_id":91,"id":4414870,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'failed'.","timestamp":"2025-11-05 01:49:36","update_id":778301,"user_id":91,"id":4414871,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":-1,"karma_critpath":0,"text":"As the openQA failure indicates, grub2 does not create a proper Grub menu and the only one item to select is the UEFI menu, which is not what we expect in the tests.\n\nhttps://openqa.fedoraproject.org/tests/3939191#step/_console_wait_login_2/3","timestamp":"2025-11-06 13:55:06","update_id":778301,"user_id":4049,"id":4417096,"bug_feedback":[{"karma":0,"comment_id":4417096,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4417096,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lruzicka","email":"lruzicka@redhat.com","id":4049,"avatar":null,"openid":null,"groups":[{"name":"qa-tools-sig"},{"name":"qa"},{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"},{"name":"sysadmin"},{"name":"sysadmin-qa"},{"name":"qa-admin"},{"name":"common-issues-triage"}]}},{"karma":0,"karma_critpath":0,"text":"I believe you, but... i didn't touch the grub2 code, this is just a rebuild against the newer fuse3 API. Can you open a bug against grub2?","timestamp":"2025-11-06 14:53:46","update_id":778301,"user_id":224,"id":4417155,"bug_feedback":[{"karma":0,"comment_id":4417155,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4417155,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"spot","email":"spotrh@gmail.com","id":224,"avatar":null,"openid":null,"groups":[{"name":"provenpackager"},{"name":"packager"},{"name":"arm-qa"},{"name":"cvsfedora"},{"name":"mirror-ia64"},{"name":"gitfedora-logos"},{"name":"fedorabugs"},{"name":"ambassadors"},{"name":"signed_fpca"},{"name":"openhw2012"},{"name":"paste-deleter"},{"name":"3d-printing-sig"},{"name":"cla_redhat"},{"name":"fedora-contributor"},{"name":"gitgeneric-release"},{"name":"gitmoksha"},{"name":"irc-support-operators"},{"name":"ipausers"},{"name":"r-maint-sig"}]}},{"karma":0,"karma_critpath":0,"text":"Isn't grub2 one of those special packages that only select few people can build \"correctly\", otherwise it won't get signed for secure boot?","timestamp":"2025-11-06 19:41:09","update_id":778301,"user_id":388,"id":4417381,"bug_feedback":[{"karma":0,"comment_id":4417381,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4417381,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"decathorpe","email":"decathorpe@gmail.com","id":388,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"fedorabugs"},{"name":"fedora-contributor"},{"name":"packaging-committee"},{"name":"pantheon-sig"},{"name":"multimedia-sig"},{"name":"ipausers"},{"name":"libreoffice-sig"},{"name":"rust-sig"},{"name":"diversity-pride"},{"name":"signed_fpca"},{"name":"fesco"},{"name":"cosmic-sig"},{"name":"gnome-sig"},{"name":"provenpackager"}]}},{"karma":-1,"karma_critpath":0,"text":"Yes it is, but I don't think that problem would manifest this way. I can't easily tell whether @spot has the necessary permission.\n\n@spot , *you* may not have touched the grub2 code, but it was touched. A lot. The previously tagged build was [grub2-2.12-41.fc44](https://koji.fedoraproject.org/koji/buildinfo?buildID=2804120), which was built from e02b6e88c71642ecc063ae7aefbd09b9be80d50e . Since then, there have been five committed-but-not-built changes by @lsandoval , @nfrayer and @jstancek :\n\n* [55-set-boot-entry.install: move to 95](https://src.fedoraproject.org/rpms/grub2/c/f6bc25c3e72a962cb59f568ec5fcc4dd374a9be2?branch=rawhide)\n* [gen_grub_cfgstub: Removed output messages](https://src.fedoraproject.org/rpms/grub2/c/730ce277473cc673793f79437426c1403ea47621?branch=rawhide)\n* [Include function name on debug and error print functions](https://src.fedoraproject.org/rpms/grub2/c/b9b1d1e384bc49e29f4398d0e3ac176c584684a0?branch=rawhide)\n* [Include upstream blsuki related patches](https://src.fedoraproject.org/rpms/grub2/c/4614a09b9fba8c2ebd3f52f39d9a49ecb12c4c59?branch=rawhide)\n* [Fix blsuki patch enumeration](https://src.fedoraproject.org/rpms/grub2/c/f4663b0ddc6e7b05b81bff630061838b77e725bc?branch=rawhide)\n\nso the problem is likely in one of those. grub maintainers, can you please avoid unbuilt commits? It causes problems...exactly like this.","timestamp":"2025-11-10 17:34:49","update_id":778301,"user_id":302,"id":4421987,"bug_feedback":[{"karma":0,"comment_id":4421987,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4421987,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"@lsandova\n\ncan't find the right FAS name for jstancek.","timestamp":"2025-11-10 17:35:32","update_id":778301,"user_id":302,"id":4421989,"bug_feedback":[{"karma":0,"comment_id":4421989,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4421989,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"The serial console output fortunately captures this error, which doesn't appear in the pre-update boot, or in passed tests:\n```\n00BdsDxe: loading Boot0003 \"Fedora\" from HD(1,GPT,FCCD1E98-E89E-4571-A00F-02D3BB70E0D4,0x800,0x12C000)/\\EFI\\fedora\\shimx64.efi\nBdsDxe: starting Boot0003 \"Fedora\" from HD(1,GPT,FCCD1E98-E89E-4571-A00F-02D3BB70E0D4,0x800,0x12C000)/\\EFI\\fedora\\shimx64.efi\nerror: ../../grub-core/kern/misc.c:grub_strtoull:606:unrecognized number.\n\nGRUB version 2.12\n```","timestamp":"2025-11-10 17:39:29","update_id":778301,"user_id":302,"id":4421996,"bug_feedback":[{"karma":0,"comment_id":4421996,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4421996,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"BTW, rmdepcheck also found some things that you missed rebuilding, @spot :\n```\nDependencies of other packages that would be BROKEN by the tested packages:\npackage: apfs-fuse-0-30.20200928gitee71aa5.fc43.x86_64 from https://kojipkgs.fedoraproject.org/repos/f44-build/latest/x86_64\n libfuse3.so.3()(64bit)\n libfuse3.so.3(FUSE_3.0)(64bit)\npackage: bindfs-1.18.3-1.fc44.x86_64 from https://kojipkgs.fedoraproject.org/repos/f44-build/latest/x86_64\n libfuse3.so.3()(64bit)\n libfuse3.so.3(FUSE_3.0)(64bit)\n```","timestamp":"2025-11-10 18:47:04","update_id":778301,"user_id":302,"id":4422036,"bug_feedback":[{"karma":0,"comment_id":4422036,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4422036,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"spot edited this update.\n\nNew build(s):\n\n- apfs-fuse-0-31.20200928gitee71aa5.fc44\n- bindfs-1.18.3-2.fc44\n\nKarma has been reset.","timestamp":"2025-11-11 14:15:48","update_id":778301,"user_id":91,"id":4422404,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"Added the missing deps. Still not touching grub.","timestamp":"2025-11-11 14:16:40","update_id":778301,"user_id":224,"id":4422405,"bug_feedback":[{"karma":0,"comment_id":4422405,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4422405,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"spot","email":"spotrh@gmail.com","id":224,"avatar":null,"openid":null,"groups":[{"name":"provenpackager"},{"name":"packager"},{"name":"arm-qa"},{"name":"cvsfedora"},{"name":"mirror-ia64"},{"name":"gitfedora-logos"},{"name":"fedorabugs"},{"name":"ambassadors"},{"name":"signed_fpca"},{"name":"openhw2012"},{"name":"paste-deleter"},{"name":"3d-printing-sig"},{"name":"cla_redhat"},{"name":"fedora-contributor"},{"name":"gitgeneric-release"},{"name":"gitmoksha"},{"name":"irc-support-operators"},{"name":"ipausers"},{"name":"r-maint-sig"}]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'waiting'.","timestamp":"2025-11-11 14:17:44","update_id":778301,"user_id":91,"id":4422406,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'failed'.","timestamp":"2025-11-11 14:31:48","update_id":778301,"user_id":91,"id":4422411,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'waiting'.","timestamp":"2025-11-11 15:22:43","update_id":778301,"user_id":91,"id":4422480,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'failed'.","timestamp":"2025-11-11 15:22:56","update_id":778301,"user_id":91,"id":4422481,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"thanks. indeed grub still broken. Guess we need the maintainers to figure out what they broke. Might need to ping them internally, not sure they read bodhi mail.","timestamp":"2025-11-11 17:05:08","update_id":778301,"user_id":302,"id":4422564,"bug_feedback":[{"karma":0,"comment_id":4422564,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4422564,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"Note this update also has at least one conflict - gvfs - with the [massive ffmpeg 8 rebuild](https://bodhi.fedoraproject.org/updates/FEDORA-2025-2a86d059f0) that is currently in-flight. that one *should* land soon, so anything in that update that is also in this update will need to be re-done in this update so it's built against both ffmpeg 8 *and* fuse 3.17.4.","timestamp":"2025-11-12 19:02:42","update_id":778301,"user_id":302,"id":4424246,"bug_feedback":[{"karma":0,"comment_id":4424246,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4424246,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"@adamwill sorry for missing the notifications on this one. Let me investigate what's going on","timestamp":"2025-11-12 19:30:56","update_id":778301,"user_id":8326,"id":4424259,"bug_feedback":[{"karma":0,"comment_id":4424259,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4424259,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lsandova","email":"lsandova@redhat.com","id":8326,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"}]}},{"karma":0,"karma_critpath":0,"text":"adamwill edited this update.\n\nNew build(s):\n\n- gvfs-1.58.0-4.fc44\n\nRemoved build(s):\n\n- gvfs-1.58.0-2.fc44\n\nKarma has been reset.","timestamp":"2025-11-12 20:02:55","update_id":778301,"user_id":91,"id":4424273,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"I did a new gvfs build.","timestamp":"2025-11-12 20:03:14","update_id":778301,"user_id":302,"id":4424274,"bug_feedback":[{"karma":0,"comment_id":4424274,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4424274,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"@adamwill, as you pointed out, we missed several unbuilds commits but in case there is mundane reason[*]. BTW, scratch builds work fine [1] so this is confusing.\n\nBelow are the related commits and perhaps 2.12-42 or 2.12-46 may have introduced this issue but I am not seeing immediately how. The rest of the patches are not related at all but of course I have to prove it.\n\nBesides the debug log provided in previous comments, is there other serial/console log to look?\n\n* Tue Nov 04 2025 Tom Callaway - 2.12-46\n- rebuild for new fuse3\n\n* Wed Oct 15 2025 Leo Sandoval - 2.12-45\n- Include upstream blsuki related patches\n\n* Mon Sep 15 2025 Leo Sandoval - 2.12-44\n- Include function name on debug print function\n\n* Fri Aug 29 2025 Nicolas Frayer - 2.12-43\n- gen_grub_cfgstub: Removed output messages\n\n* Wed Aug 20 2025 Jan Stancek - 2.12-42\n- move 55-set-boot-entry.install to 95\n- Related: #2389020\n\n[*] my fedpkg build hangs at some point and perhaps this has to do with my IP provider not support IPv6\n\n[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=138860636","timestamp":"2025-11-12 20:04:00","update_id":778301,"user_id":8326,"id":4424275,"bug_feedback":[{"karma":0,"comment_id":4424275,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4424275,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lsandova","email":"lsandova@redhat.com","id":8326,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"}]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'waiting'.","timestamp":"2025-11-12 20:04:39","update_id":778301,"user_id":91,"id":4424276,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'failed'.","timestamp":"2025-11-12 20:18:53","update_id":778301,"user_id":91,"id":4424282,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"All we really know is that after we install the grub2 update and reboot, the system (a UEFI VM) boots to the grub menu but the grub menu does not contain the expected boot menu entry for booting Fedora, instead it contains only the menu entry for running the UEFI firmware setup. And there's that `strtoull` error in the serial console log.\n\nI can hack the test up to check whether the Fedora kernel entry is actually there in the grub/bls config - I suspect it *is* and the problem is that grub is choking on parsing it for some reason so it's not showing it, but I haven't checked for sure.\n\nBeyond that, you'd have to let us know how to get grub to output more verbose debugging info, then we can hack up a test to do that. We can also try to reproduce in a local VM.","timestamp":"2025-11-12 21:02:32","update_id":778301,"user_id":302,"id":4424371,"bug_feedback":[{"karma":0,"comment_id":4424371,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4424371,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"adamwill edited this update.\n\nNew build(s):\n\n- flatpak-1.16.1-10.fc44\n\nRemoved build(s):\n\n- flatpak-1.16.1-6.fc44\n\nKarma has been reset.","timestamp":"2025-11-12 21:42:26","update_id":778301,"user_id":91,"id":4424390,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'waiting'.","timestamp":"2025-11-12 21:44:12","update_id":778301,"user_id":91,"id":4424396,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'failed'.","timestamp":"2025-11-12 21:44:14","update_id":778301,"user_id":91,"id":4424397,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'waiting'.","timestamp":"2025-11-12 21:50:38","update_id":778301,"user_id":91,"id":4424401,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"This update's test gating status has been changed to 'failed'.","timestamp":"2025-11-12 21:59:32","update_id":778301,"user_id":91,"id":4424420,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}},{"karma":0,"karma_critpath":0,"text":"OK, so I can reproduce this in a local UEFI (non-SB) VM. Just do a minimal Rawhide install, download the update packages, update, and reboot.\n\nThe grub config file has the `blscfg` line which should generate entries from the BLS snippets, but obviously...isn't, somehow. The BLS snippets are there.\n\nI can get to the grub console, but no idea what to do there to diagnose the problem. If I run `blscfg` nothing appears to happen - there's no output, either good or bad.\n\nPlease let me know what I can do to help diagnose the problem.","timestamp":"2025-11-12 22:31:04","update_id":778301,"user_id":302,"id":4424457,"bug_feedback":[{"karma":0,"comment_id":4424457,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4424457,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}},{"karma":0,"karma_critpath":0,"text":"@adamwill\n\nlet me start debugging. \n\nBTW, in case you want to debug it further, on a running system, at the prompt, type\n\n$ grub2-editenv - set debug=all,-lexer,-scripting\n\nthen reboot. That indicates high verbosity except the parsing stuff. Expect lots of log.","timestamp":"2025-11-13 15:46:03","update_id":778301,"user_id":8326,"id":4426138,"bug_feedback":[{"karma":0,"comment_id":4426138,"bug_id":2346446,"bug":{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false}}],"testcase_feedback":[{"karma":0,"comment_id":4426138,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lsandova","email":"lsandova@redhat.com","id":8326,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"}]}},{"karma":0,"karma_critpath":0,"text":"spot edited this update.\n\nNew build(s):\n\n- flatpak-1.16.1-13.fc44\n\nRemoved build(s):\n\n- flatpak-1.16.1-10.fc44\n\nKarma has been reset.","timestamp":"2025-11-13 17:11:49","update_id":778301,"user_id":91,"id":4426180,"bug_feedback":[],"testcase_feedback":[],"user":{"name":"bodhi","email":null,"id":91,"avatar":null,"openid":null,"groups":[]}}],"builds":[{"nvr":"android-file-transfer-4.5-4.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"apfs-fuse-0-31.20200928gitee71aa5.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"apptainer-1.4.4-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"bindfs-1.18.3-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"btrfs-fuse-0-19.20230119git812c4b7.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"buildbox-1.3.42-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"ceph-20.1.1-5.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"charliecloud-0.42-1.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"e2fsprogs-1.47.3-3.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"erofs-utils-1.8.10-3.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"freerdp-3.17.2-3.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"fuse3-3.17.4-1.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"fuse-btfs-3.1-3.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"fuse-overlayfs-1.13-5.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"fuse-sshfs-3.7.3-13.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"gnome-connections-49.0-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"gnome-remote-desktop-49.1-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"grub2-2.12-46.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"gvfs-1.58.0-4.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"kio-fuse-5.1.1-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"libnbd-1.23.10-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"libvirt-11.9.0-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"open-vm-tools-13.0.0-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"ostree-2025.6-3.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"partclone-0.3.38-2.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"qemu-10.1.2-4.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"wimlib-1.14.4-6.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"xdg-desktop-portal-1.20.3-4.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"xmount-1.2.0-5.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"xrdp-0.10.4-4.fc44","signed":true,"release_id":93,"type":"rpm","epoch":0},{"nvr":"flatpak-1.16.1-13.fc44","signed":false,"release_id":93,"type":"rpm","epoch":0}],"bugs":[{"bug_id":2346446,"title":"fuse3-3.17.4 is available","security":false,"parent":false,"feedback":[{"karma":0,"comment_id":4417096,"bug_id":2346446,"comment":{"karma":-1,"karma_critpath":0,"text":"As the openQA failure indicates, grub2 does not create a proper Grub menu and the only one item to select is the UEFI menu, which is not what we expect in the tests.\n\nhttps://openqa.fedoraproject.org/tests/3939191#step/_console_wait_login_2/3","timestamp":"2025-11-06 13:55:06","update_id":778301,"user_id":4049,"id":4417096,"testcase_feedback":[{"karma":0,"comment_id":4417096,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lruzicka","email":"lruzicka@redhat.com","id":4049,"avatar":null,"openid":null,"groups":[{"name":"qa-tools-sig"},{"name":"qa"},{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"},{"name":"sysadmin"},{"name":"sysadmin-qa"},{"name":"qa-admin"},{"name":"common-issues-triage"}]}}},{"karma":0,"comment_id":4417155,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"I believe you, but... i didn't touch the grub2 code, this is just a rebuild against the newer fuse3 API. Can you open a bug against grub2?","timestamp":"2025-11-06 14:53:46","update_id":778301,"user_id":224,"id":4417155,"testcase_feedback":[{"karma":0,"comment_id":4417155,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"spot","email":"spotrh@gmail.com","id":224,"avatar":null,"openid":null,"groups":[{"name":"provenpackager"},{"name":"packager"},{"name":"arm-qa"},{"name":"cvsfedora"},{"name":"mirror-ia64"},{"name":"gitfedora-logos"},{"name":"fedorabugs"},{"name":"ambassadors"},{"name":"signed_fpca"},{"name":"openhw2012"},{"name":"paste-deleter"},{"name":"3d-printing-sig"},{"name":"cla_redhat"},{"name":"fedora-contributor"},{"name":"gitgeneric-release"},{"name":"gitmoksha"},{"name":"irc-support-operators"},{"name":"ipausers"},{"name":"r-maint-sig"}]}}},{"karma":0,"comment_id":4417381,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"Isn't grub2 one of those special packages that only select few people can build \"correctly\", otherwise it won't get signed for secure boot?","timestamp":"2025-11-06 19:41:09","update_id":778301,"user_id":388,"id":4417381,"testcase_feedback":[{"karma":0,"comment_id":4417381,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"decathorpe","email":"decathorpe@gmail.com","id":388,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"fedorabugs"},{"name":"fedora-contributor"},{"name":"packaging-committee"},{"name":"pantheon-sig"},{"name":"multimedia-sig"},{"name":"ipausers"},{"name":"libreoffice-sig"},{"name":"rust-sig"},{"name":"diversity-pride"},{"name":"signed_fpca"},{"name":"fesco"},{"name":"cosmic-sig"},{"name":"gnome-sig"},{"name":"provenpackager"}]}}},{"karma":0,"comment_id":4421987,"bug_id":2346446,"comment":{"karma":-1,"karma_critpath":0,"text":"Yes it is, but I don't think that problem would manifest this way. I can't easily tell whether @spot has the necessary permission.\n\n@spot , *you* may not have touched the grub2 code, but it was touched. A lot. The previously tagged build was [grub2-2.12-41.fc44](https://koji.fedoraproject.org/koji/buildinfo?buildID=2804120), which was built from e02b6e88c71642ecc063ae7aefbd09b9be80d50e . Since then, there have been five committed-but-not-built changes by @lsandoval , @nfrayer and @jstancek :\n\n* [55-set-boot-entry.install: move to 95](https://src.fedoraproject.org/rpms/grub2/c/f6bc25c3e72a962cb59f568ec5fcc4dd374a9be2?branch=rawhide)\n* [gen_grub_cfgstub: Removed output messages](https://src.fedoraproject.org/rpms/grub2/c/730ce277473cc673793f79437426c1403ea47621?branch=rawhide)\n* [Include function name on debug and error print functions](https://src.fedoraproject.org/rpms/grub2/c/b9b1d1e384bc49e29f4398d0e3ac176c584684a0?branch=rawhide)\n* [Include upstream blsuki related patches](https://src.fedoraproject.org/rpms/grub2/c/4614a09b9fba8c2ebd3f52f39d9a49ecb12c4c59?branch=rawhide)\n* [Fix blsuki patch enumeration](https://src.fedoraproject.org/rpms/grub2/c/f4663b0ddc6e7b05b81bff630061838b77e725bc?branch=rawhide)\n\nso the problem is likely in one of those. grub maintainers, can you please avoid unbuilt commits? It causes problems...exactly like this.","timestamp":"2025-11-10 17:34:49","update_id":778301,"user_id":302,"id":4421987,"testcase_feedback":[{"karma":0,"comment_id":4421987,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4421989,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"@lsandova\n\ncan't find the right FAS name for jstancek.","timestamp":"2025-11-10 17:35:32","update_id":778301,"user_id":302,"id":4421989,"testcase_feedback":[{"karma":0,"comment_id":4421989,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4421996,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"The serial console output fortunately captures this error, which doesn't appear in the pre-update boot, or in passed tests:\n```\n00BdsDxe: loading Boot0003 \"Fedora\" from HD(1,GPT,FCCD1E98-E89E-4571-A00F-02D3BB70E0D4,0x800,0x12C000)/\\EFI\\fedora\\shimx64.efi\nBdsDxe: starting Boot0003 \"Fedora\" from HD(1,GPT,FCCD1E98-E89E-4571-A00F-02D3BB70E0D4,0x800,0x12C000)/\\EFI\\fedora\\shimx64.efi\nerror: ../../grub-core/kern/misc.c:grub_strtoull:606:unrecognized number.\n\nGRUB version 2.12\n```","timestamp":"2025-11-10 17:39:29","update_id":778301,"user_id":302,"id":4421996,"testcase_feedback":[{"karma":0,"comment_id":4421996,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4422036,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"BTW, rmdepcheck also found some things that you missed rebuilding, @spot :\n```\nDependencies of other packages that would be BROKEN by the tested packages:\npackage: apfs-fuse-0-30.20200928gitee71aa5.fc43.x86_64 from https://kojipkgs.fedoraproject.org/repos/f44-build/latest/x86_64\n libfuse3.so.3()(64bit)\n libfuse3.so.3(FUSE_3.0)(64bit)\npackage: bindfs-1.18.3-1.fc44.x86_64 from https://kojipkgs.fedoraproject.org/repos/f44-build/latest/x86_64\n libfuse3.so.3()(64bit)\n libfuse3.so.3(FUSE_3.0)(64bit)\n```","timestamp":"2025-11-10 18:47:04","update_id":778301,"user_id":302,"id":4422036,"testcase_feedback":[{"karma":0,"comment_id":4422036,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4422405,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"Added the missing deps. Still not touching grub.","timestamp":"2025-11-11 14:16:40","update_id":778301,"user_id":224,"id":4422405,"testcase_feedback":[{"karma":0,"comment_id":4422405,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"spot","email":"spotrh@gmail.com","id":224,"avatar":null,"openid":null,"groups":[{"name":"provenpackager"},{"name":"packager"},{"name":"arm-qa"},{"name":"cvsfedora"},{"name":"mirror-ia64"},{"name":"gitfedora-logos"},{"name":"fedorabugs"},{"name":"ambassadors"},{"name":"signed_fpca"},{"name":"openhw2012"},{"name":"paste-deleter"},{"name":"3d-printing-sig"},{"name":"cla_redhat"},{"name":"fedora-contributor"},{"name":"gitgeneric-release"},{"name":"gitmoksha"},{"name":"irc-support-operators"},{"name":"ipausers"},{"name":"r-maint-sig"}]}}},{"karma":0,"comment_id":4422564,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"thanks. indeed grub still broken. Guess we need the maintainers to figure out what they broke. Might need to ping them internally, not sure they read bodhi mail.","timestamp":"2025-11-11 17:05:08","update_id":778301,"user_id":302,"id":4422564,"testcase_feedback":[{"karma":0,"comment_id":4422564,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4424246,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"Note this update also has at least one conflict - gvfs - with the [massive ffmpeg 8 rebuild](https://bodhi.fedoraproject.org/updates/FEDORA-2025-2a86d059f0) that is currently in-flight. that one *should* land soon, so anything in that update that is also in this update will need to be re-done in this update so it's built against both ffmpeg 8 *and* fuse 3.17.4.","timestamp":"2025-11-12 19:02:42","update_id":778301,"user_id":302,"id":4424246,"testcase_feedback":[{"karma":0,"comment_id":4424246,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4424259,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"@adamwill sorry for missing the notifications on this one. Let me investigate what's going on","timestamp":"2025-11-12 19:30:56","update_id":778301,"user_id":8326,"id":4424259,"testcase_feedback":[{"karma":0,"comment_id":4424259,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lsandova","email":"lsandova@redhat.com","id":8326,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"}]}}},{"karma":0,"comment_id":4424274,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"I did a new gvfs build.","timestamp":"2025-11-12 20:03:14","update_id":778301,"user_id":302,"id":4424274,"testcase_feedback":[{"karma":0,"comment_id":4424274,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4424275,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"@adamwill, as you pointed out, we missed several unbuilds commits but in case there is mundane reason[*]. BTW, scratch builds work fine [1] so this is confusing.\n\nBelow are the related commits and perhaps 2.12-42 or 2.12-46 may have introduced this issue but I am not seeing immediately how. The rest of the patches are not related at all but of course I have to prove it.\n\nBesides the debug log provided in previous comments, is there other serial/console log to look?\n\n* Tue Nov 04 2025 Tom Callaway - 2.12-46\n- rebuild for new fuse3\n\n* Wed Oct 15 2025 Leo Sandoval - 2.12-45\n- Include upstream blsuki related patches\n\n* Mon Sep 15 2025 Leo Sandoval - 2.12-44\n- Include function name on debug print function\n\n* Fri Aug 29 2025 Nicolas Frayer - 2.12-43\n- gen_grub_cfgstub: Removed output messages\n\n* Wed Aug 20 2025 Jan Stancek - 2.12-42\n- move 55-set-boot-entry.install to 95\n- Related: #2389020\n\n[*] my fedpkg build hangs at some point and perhaps this has to do with my IP provider not support IPv6\n\n[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=138860636","timestamp":"2025-11-12 20:04:00","update_id":778301,"user_id":8326,"id":4424275,"testcase_feedback":[{"karma":0,"comment_id":4424275,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lsandova","email":"lsandova@redhat.com","id":8326,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"}]}}},{"karma":0,"comment_id":4424371,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"All we really know is that after we install the grub2 update and reboot, the system (a UEFI VM) boots to the grub menu but the grub menu does not contain the expected boot menu entry for booting Fedora, instead it contains only the menu entry for running the UEFI firmware setup. And there's that `strtoull` error in the serial console log.\n\nI can hack the test up to check whether the Fedora kernel entry is actually there in the grub/bls config - I suspect it *is* and the problem is that grub is choking on parsing it for some reason so it's not showing it, but I haven't checked for sure.\n\nBeyond that, you'd have to let us know how to get grub to output more verbose debugging info, then we can hack up a test to do that. We can also try to reproduce in a local VM.","timestamp":"2025-11-12 21:02:32","update_id":778301,"user_id":302,"id":4424371,"testcase_feedback":[{"karma":0,"comment_id":4424371,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4424457,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"OK, so I can reproduce this in a local UEFI (non-SB) VM. Just do a minimal Rawhide install, download the update packages, update, and reboot.\n\nThe grub config file has the `blscfg` line which should generate entries from the BLS snippets, but obviously...isn't, somehow. The BLS snippets are there.\n\nI can get to the grub console, but no idea what to do there to diagnose the problem. If I run `blscfg` nothing appears to happen - there's no output, either good or bad.\n\nPlease let me know what I can do to help diagnose the problem.","timestamp":"2025-11-12 22:31:04","update_id":778301,"user_id":302,"id":4424457,"testcase_feedback":[{"karma":0,"comment_id":4424457,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"adamwill","email":"awilliam@redhat.com","id":302,"avatar":null,"openid":null,"groups":[{"name":"proventesters"},{"name":"provenpackager"},{"name":"packager"},{"name":"qa-tools-sig"},{"name":"fedora-contributor"},{"name":"news"},{"name":"gittriage"},{"name":"sysadmin"},{"name":"gitspin-kickstarts"},{"name":"signed_fpca"},{"name":"gitgeneric-release"},{"name":"yak_farmers"},{"name":"triagers"},{"name":"qa"},{"name":"sysadmin-qa"},{"name":"fedorabugs"},{"name":"ipausers"},{"name":"gitmkinitrd"},{"name":"qa-admin"},{"name":"server-wg"},{"name":"forge-cle-members"},{"name":"aws-qa"},{"name":"change-wranglers"},{"name":"fedora-ci-admins"},{"name":"common-issues-triage"},{"name":"sysadmin-main"},{"name":"program-management"},{"name":"gitfedora-project-schedule"},{"name":"forge-atomic-desktops-members"}]}}},{"karma":0,"comment_id":4426138,"bug_id":2346446,"comment":{"karma":0,"karma_critpath":0,"text":"@adamwill\n\nlet me start debugging. \n\nBTW, in case you want to debug it further, on a running system, at the prompt, type\n\n$ grub2-editenv - set debug=all,-lexer,-scripting\n\nthen reboot. That indicates high verbosity except the parsing stuff. Expect lots of log.","timestamp":"2025-11-13 15:46:03","update_id":778301,"user_id":8326,"id":4426138,"testcase_feedback":[{"karma":0,"comment_id":4426138,"testcase_id":575,"testcase":{"name":"QA:Testcase e2fsprogs","id":575}}],"user":{"name":"lsandova","email":"lsandova@redhat.com","id":8326,"avatar":null,"openid":null,"groups":[{"name":"packager"},{"name":"ipausers"},{"name":"fedora-contributor"},{"name":"signed_fpca"},{"name":"fedorabugs"}]}}}]}],"updateid":"FEDORA-2025-fb66eccf2c","karma":0,"content_type":"rpm","test_cases":[{"name":"QA:Testcase e2fsprogs","id":575}]}} [2025-11-13T17:11:56.737Z] [Pipeline] readJSON [2025-11-13T17:11:56.876Z] [Pipeline] libraryResource [2025-11-13T17:11:56.888Z] [Pipeline] readJSON [2025-11-13T17:11:56.935Z] [Pipeline] node [2025-11-13T17:11:56.950Z] Running on Jenkins in /var/lib/jenkins/jobs/bodhi-trigger/workspace [2025-11-13T17:11:56.984Z] [Pipeline] { [2025-11-13T17:11:57.032Z] [Pipeline] writeYaml [2025-11-13T17:11:57.074Z] [Pipeline] readFile [2025-11-13T17:11:57.114Z] [Pipeline] } [2025-11-13T17:11:57.194Z] [Pipeline] // node [2025-11-13T17:11:57.214Z] [Pipeline] podTemplate [2025-11-13T17:11:57.218Z] [Pipeline] { [2025-11-13T17:11:57.259Z] [Pipeline] node [2025-11-13T17:11:58.289Z] Created Pod: openshift coreos-ci/pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5 [2025-11-13T17:11:58.317Z] [PodInfo] coreos-ci/pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5 [2025-11-13T17:11:58.317Z] Container [jnlp] waiting [ContainerCreating] No message [2025-11-13T17:11:58.317Z] Container [worker] waiting [ContainerCreating] No message [2025-11-13T17:11:58.317Z] Pod [Pending][ContainersNotReady] containers with unready status: [worker jnlp] [2025-11-13T17:11:58.662Z] [PodInfo] coreos-ci/pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5 [2025-11-13T17:11:58.662Z] Container [jnlp] waiting [ContainerCreating] No message [2025-11-13T17:11:58.662Z] Container [worker] waiting [ContainerCreating] No message [2025-11-13T17:11:58.662Z] Pod [Pending][ContainersNotReady] containers with unready status: [worker jnlp] [2025-11-13T17:11:58.991Z] [PodInfo] coreos-ci/pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5 [2025-11-13T17:11:58.991Z] Container [jnlp] waiting [ContainerCreating] No message [2025-11-13T17:11:58.991Z] Container [worker] waiting [ContainerCreating] No message [2025-11-13T17:11:58.991Z] Pod [Pending][ContainersNotReady] containers with unready status: [worker jnlp] [2025-11-13T17:12:08.066Z] Agent pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5 is provisioned from template pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc [2025-11-13T17:12:08.067Z] --- [2025-11-13T17:12:08.067Z] apiVersion: "v1" [2025-11-13T17:12:08.067Z] kind: "Pod" [2025-11-13T17:12:08.067Z] metadata: [2025-11-13T17:12:08.067Z] annotations: [2025-11-13T17:12:08.067Z] kubernetes.jenkins.io/last-refresh: "1763053918261" [2025-11-13T17:12:08.067Z] buildUrl: "http://172.30.177.66:80/job/bodhi-trigger/410/" [2025-11-13T17:12:08.067Z] runUrl: "job/bodhi-trigger/410/" [2025-11-13T17:12:08.067Z] labels: [2025-11-13T17:12:08.067Z] jenkins: "slave" [2025-11-13T17:12:08.067Z] jenkins/label-digest: "33c7962bb796b46130d5bda075557f7a6c5031ee" [2025-11-13T17:12:08.067Z] jenkins/label: "pod-819252a1-847c-4d74-8112-969b68deae0d" [2025-11-13T17:12:08.067Z] kubernetes.jenkins.io/controller: "http___172_30_177_66_80x" [2025-11-13T17:12:08.067Z] name: "pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5" [2025-11-13T17:12:08.067Z] namespace: "coreos-ci" [2025-11-13T17:12:08.067Z] spec: [2025-11-13T17:12:08.067Z] containers: [2025-11-13T17:12:08.067Z] - command: [2025-11-13T17:12:08.067Z] - "/usr/bin/dumb-init" [2025-11-13T17:12:08.067Z] - "/usr/bin/sleep" [2025-11-13T17:12:08.067Z] - "infinity" [2025-11-13T17:12:08.067Z] env: [2025-11-13T17:12:08.067Z] - name: "NCPUS" [2025-11-13T17:12:08.067Z] value: "0.1" [2025-11-13T17:12:08.067Z] image: "quay.io/coreos-assembler/coreos-assembler:main" [2025-11-13T17:12:08.067Z] imagePullPolicy: "Always" [2025-11-13T17:12:08.067Z] name: "worker" [2025-11-13T17:12:08.067Z] resources: [2025-11-13T17:12:08.067Z] limits: [2025-11-13T17:12:08.067Z] cpu: "0.1" [2025-11-13T17:12:08.067Z] requests: [2025-11-13T17:12:08.067Z] cpu: "0.1" [2025-11-13T17:12:08.067Z] volumeMounts: [2025-11-13T17:12:08.067Z] - mountPath: "/srv/" [2025-11-13T17:12:08.067Z] name: "emptydir-0" [2025-11-13T17:12:08.067Z] - mountPath: "/home/jenkins/agent" [2025-11-13T17:12:08.067Z] name: "workspace-volume" [2025-11-13T17:12:08.067Z] readOnly: false [2025-11-13T17:12:08.067Z] - env: [2025-11-13T17:12:08.067Z] - name: "JENKINS_SECRET" [2025-11-13T17:12:08.067Z] value: "********" [2025-11-13T17:12:08.067Z] - name: "JENKINS_TUNNEL" [2025-11-13T17:12:08.067Z] value: "172.30.178.247:50000" [2025-11-13T17:12:08.067Z] - name: "JENKINS_AGENT_NAME" [2025-11-13T17:12:08.067Z] value: "pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5" [2025-11-13T17:12:08.067Z] - name: "REMOTING_OPTS" [2025-11-13T17:12:08.067Z] value: "-noReconnectAfter 1d" [2025-11-13T17:12:08.067Z] - name: "JENKINS_NAME" [2025-11-13T17:12:08.067Z] value: "pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5" [2025-11-13T17:12:08.067Z] - name: "JENKINS_AGENT_WORKDIR" [2025-11-13T17:12:08.067Z] value: "/home/jenkins/agent" [2025-11-13T17:12:08.067Z] - name: "JENKINS_URL" [2025-11-13T17:12:08.067Z] value: "http://172.30.177.66:80/" [2025-11-13T17:12:08.067Z] image: "jenkins-agent-base:latest" [2025-11-13T17:12:08.067Z] name: "jnlp" [2025-11-13T17:12:08.067Z] resources: [2025-11-13T17:12:08.067Z] limits: [2025-11-13T17:12:08.067Z] memory: "512Mi" [2025-11-13T17:12:08.067Z] cpu: "1" [2025-11-13T17:12:08.067Z] requests: [2025-11-13T17:12:08.067Z] memory: "512Mi" [2025-11-13T17:12:08.067Z] cpu: "1" [2025-11-13T17:12:08.067Z] volumeMounts: [2025-11-13T17:12:08.067Z] - mountPath: "/home/jenkins/agent" [2025-11-13T17:12:08.067Z] name: "workspace-volume" [2025-11-13T17:12:08.067Z] readOnly: false [2025-11-13T17:12:08.067Z] nodeSelector: [2025-11-13T17:12:08.067Z] kubernetes.io/os: "linux" [2025-11-13T17:12:08.067Z] restartPolicy: "Never" [2025-11-13T17:12:08.067Z] serviceAccountName: "default" [2025-11-13T17:12:08.067Z] volumes: [2025-11-13T17:12:08.067Z] - emptyDir: {} [2025-11-13T17:12:08.067Z] name: "emptydir-0" [2025-11-13T17:12:08.067Z] - emptyDir: [2025-11-13T17:12:08.067Z] medium: "" [2025-11-13T17:12:08.067Z] name: "workspace-volume" [2025-11-13T17:12:08.067Z] [2025-11-13T17:12:08.157Z] Running on pod-819252a1-847c-4d74-8112-969b68deae0d-3lplc-653t5 in /home/jenkins/agent/workspace/bodhi-trigger [2025-11-13T17:12:08.163Z] [Pipeline] { [2025-11-13T17:12:08.217Z] [Pipeline] container [2025-11-13T17:12:08.223Z] [Pipeline] { [2025-11-13T17:12:08.277Z] [Pipeline] withEnv [2025-11-13T17:12:08.281Z] [Pipeline] { [2025-11-13T17:12:08.325Z] [Pipeline] sh [2025-11-13T17:12:09.267Z] + set -xeuo pipefail [2025-11-13T17:12:09.267Z] ++ umask [2025-11-13T17:12:09.267Z] + '[' 0022 = 0000 ']' [2025-11-13T17:12:09.267Z] + cat /cosa/coreos-assembler-git.json [2025-11-13T17:12:09.267Z] + tee coreos-assembler-git.json [2025-11-13T17:12:09.267Z] { [2025-11-13T17:12:09.267Z] "date": "2025-11-11T21:38:38Z", [2025-11-13T17:12:09.267Z] "git": { [2025-11-13T17:12:09.267Z] "commit": "fb6410e1415f2f73dc57a97dc82a22283cbf20f6", [2025-11-13T17:12:09.267Z] "origin": "https://github.com/coreos/coreos-assembler.git", [2025-11-13T17:12:09.267Z] "branch": "HEAD", [2025-11-13T17:12:09.267Z] "dirty": "false" [2025-11-13T17:12:09.267Z] }, [2025-11-13T17:12:09.267Z] "file": { [2025-11-13T17:12:09.267Z] "checksum": "306ac08f59c193aff8289f678f89c93be9de3f345cfe2eeebcf7b66ff77995fe", [2025-11-13T17:12:09.267Z] "checksum_type": "sha256", [2025-11-13T17:12:09.267Z] "format": "tar.gz", [2025-11-13T17:12:09.267Z] "name": "coreos-assembler-git.tar.gz", [2025-11-13T17:12:09.267Z] "size": "13274736" [2025-11-13T17:12:09.267Z] } [2025-11-13T17:12:09.267Z] } [2025-11-13T17:12:09.325Z] [Pipeline] } [2025-11-13T17:12:09.395Z] [Pipeline] // withEnv [2025-11-13T17:12:09.414Z] [Pipeline] archiveArtifacts [2025-11-13T17:12:09.422Z] Archiving artifacts [2025-11-13T17:12:09.970Z] [Pipeline] withEnv [2025-11-13T17:12:09.975Z] [Pipeline] { [2025-11-13T17:12:10.016Z] [Pipeline] sh [2025-11-13T17:12:10.816Z] + set -xeuo pipefail [2025-11-13T17:12:10.817Z] ++ umask [2025-11-13T17:12:11.072Z] + '[' 0022 = 0000 ']' [2025-11-13T17:12:11.072Z] + rpm -qa [2025-11-13T17:12:11.072Z] + sort [2025-11-13T17:12:14.353Z] [Pipeline] } [2025-11-13T17:12:14.419Z] [Pipeline] // withEnv [2025-11-13T17:12:14.439Z] [Pipeline] archiveArtifacts [2025-11-13T17:12:14.446Z] Archiving artifacts [2025-11-13T17:12:14.489Z] [Pipeline] withEnv [2025-11-13T17:12:14.493Z] [Pipeline] { [2025-11-13T17:12:14.537Z] [Pipeline] sh [2025-11-13T17:12:15.329Z] + set -xeuo pipefail [2025-11-13T17:12:15.329Z] ++ umask [2025-11-13T17:12:15.329Z] + '[' 0022 = 0000 ']' [2025-11-13T17:12:15.329Z] + rm coreos-assembler-git.json coreos-assembler-rpmdb.txt [2025-11-13T17:12:15.366Z] [Pipeline] } [2025-11-13T17:12:15.432Z] [Pipeline] // withEnv [2025-11-13T17:12:15.450Z] [Pipeline] stage [2025-11-13T17:12:15.453Z] [Pipeline] { (Trigger Test) [2025-11-13T17:12:15.505Z] [Pipeline] build [2025-11-13T17:12:15.510Z] Scheduling project: test-override [2025-11-13T17:12:15.519Z] [Pipeline] } [2025-11-13T17:12:15.583Z] [Pipeline] // stage [2025-11-13T17:12:15.595Z] [Pipeline] } [2025-11-13T17:12:15.669Z] [Pipeline] // container [2025-11-13T17:12:15.681Z] [Pipeline] } [2025-11-13T17:12:15.780Z] [Pipeline] // node [2025-11-13T17:12:15.793Z] [Pipeline] } [2025-11-13T17:12:15.855Z] [Pipeline] // podTemplate [2025-11-13T17:12:15.871Z] [Pipeline] End of Pipeline [2025-11-13T17:12:15.910Z] Finished: SUCCESS