This job is similar in structure to `bump-lockfile` (i.e. it runs tests on multiple arches in parallel), but is geared towards testing Bodhi and/or Koji overrides. The `OVERRIDES` parameter supports a list of URLs from those services that the job will automatically download into the `overrides/rpm` directory when doing the build.
For now, like bump-lockfile, this only does local testing, i.e. QEMU and testiso scenarios. We could extend this in the future to do some cloud testing too.
This is prep for having another job that triggers this one based on Bodhi Fedora messages, but it's structured to be useful on its own too for ad-hoc testing.
This job is a step towards better integrating our CI into the rest of Fedora QA.
The job triggers on Fedora messages from Bodhi whenever an update is created or edited. It then resolves the Bodhi update to a relevant FCOS stream, and trigger the `test-override` job with the right parameters.
The implementation details here are a bit messy. Triggering on "a new Bodhi update" isn't as simple as I would've liked and mapping back to streams is a bit hairy, but it works!
There's of course a lot of cleanups we could do here (many are marked in the code), though I'm hoping we can get this in to gain some early experience and work on cleanups in parallel with the larger task of adding reporting and socializing it.
See also: https://github.com/coreos/fedora-coreos-tracker/issues/17
Call out to the new `resultsdb-report` script in cosa to report test results back to ResultsDB. Report when first running and then again with final test results.
Just this alone seems sufficient for the results to show up in Bodhi's "Automated Tests" tab.
With this, we'll then be able to work on adding policies in Greenwave.
jenkins/config: add JCASC dropin for jms-messaging-plugin
This configures the JMS messaging plugin to connect to the public endpoint of the Fedora messaging bus.
The tricky thing is passing in the keystores. The plugin does not support using Jenkins credentials[[1]] so we can't leverage kubernetes-credentials-provider.
We could do it the old way, which is to mount the secret into the Jenkins pod, but since d6d1f61, CoreOS CI now uses the exact same `jenkins.yaml` manifest as the production pipeline and we don't want to bind mount it there.
Instead, we hack around this by just baking the keystores in the Jenkins image at `$JENKINS_HOME/jms-messaging-stores`.
jobs/test-override: on mechanical streams, build on top of known good version
When testing a rawhide update, we should be testing on top of a rawhide build that we know has worked. Otherwise, the test may fail for any arbitrary reason. Since rawhide is not locked, we do this by autolocking on top of the most recent build that contains all the relevant arches.
See also: https://github.com/coreos/fedora-coreos-releng-automation/issues/181
jobs/bodhi-trigger: read SRPM list from `bodhi-testing.yaml`
Add a new `bodhi-testing.yaml` file containing the list of SRPMs to test. We'll augment the schema of this file later on to allow e.g. running only a subset of tests.
This will also in the short-term serve as the source of truth for which packages are gated on these tests or not. Long-term, we may use this file slightly differently, though likely the tests to run per component will still live there (including the default tests to run on any unlisted component).
While we're here, add a few more core packages to the list.
bodhi-testing.yaml: make test patterns more globby
The test pattern for `toolbox` was incorrect and so didn't match anything.
For that component and some others, instead of specifying a full test path, make the patterns more globby. That makes them more obviously right, and resistant to the tests moving.
Fixes 6c1f869 ("bodhi-testing.yaml: support specifying tests to run").
The Prow CI we have in those repos are extremely slow and annoying to maintain. We're still going to need it for now to at least build RHCOS with actual RHEL RPMs, but at least for CentOS Stream we should be able to build that fine in CoreOS CI. (We don't have access to the OCP RPMs, but with https://github.com/openshift/os/issues/799, we'll move those out of the base compose anyway.)
We had it in `shwrap`, but not `shwrapCapture` and `shwrapRc`. We're hitting an issue right now where `podman` wants to create `$HOME/.ssh` when using the remote stuff but because we're running unprivileged, we get
Revert "Set `HOME` at the pod level instead of in shwrap helpers"
This reverts commit 3577892a9cac46964a5f44be8881a9ef7741df8c.
Setting the `HOME` var to the workspace from the pod definition doesn't work because it introduces a chicken-and-egg problem: the workspace isn't yet allocated since the pod isn't running yet.
This isn't a pure revert since I wanted to keep 33d4910 ("Add `umask` workaround to other shwrap helpers"), which came after.
kola: refactor and add support for warn: true tests
This commit refactors the arguments and error handling for `kola` into a runKola closure to make it easier to manage. Most calls in this file use mostly the same arguments, so this is a nice cleanup.
This commit also adds the `--on-warn-failure-exit-77` option [1] and adds handling to mark a test as `warn()` if that happens versus the typical `error()`.
Right now in the upgrade case we end up with `cosa kola run --upgrades` and that doesn't quite pass through everything right:
``` + cosa kola run --rerun --allow-rerun-success=tags=needs-internet --build=39.20230823.92.0 --output-dir=/home/jenkins/agent/workspace/kola-gcp/tmp/kola-UJ7JH/kola-upgrade --on-warn-failure-exit-77 --arch=x86_64 -p=gcp --gcp-json-key=**** --gcp-project=fedora-coreos-testing --upgrades
kola --build 39.20230823.92.0 run --rerun --allow-rerun-success=tags=needs-internet --on-warn-failure-exit-77 --arch=x86_64 -p=gcp --gcp-json-key=**** --gcp-project=fedora-coreos-testing --output-dir /home/jenkins/agent/workspace/kola-gcp/tmp/kola-UJ7JH/kola-upgrade --qemu-image-dir tmp/kola-qemu-cache -v --find-parent-image
Error: unknown flag: --qemu-image-dir ```
We need the ultimate command to be `kola run-upgrade`. Let's add the action we want as an argument to the runKola helper.
The way cosa kola parses things and then passes it on to kola is a mess. Let's remove the spaces so we can make sure the value doesn't get separated from the option when it's processed.
kola/kolaTestIso: use `kola-junit` to publish JUnit test results
This will allow us to much more easily inspect failing kola tests. Goes together with https://github.com/coreos/coreos-assembler/pull/3667, which has more details.
While we're here, use `kola-testiso` as marker since it looks clearer and for consistency with `kola-upgrade`.
![[Jenkins]](/static/ddfb44e8/images/svgs/logo.svg){kind=logo}
