Changes

jobs/bodhi-trigger: report SUCCESS even for UNSTABLE builds

When a snoozed test expires, if it has the `warn: true` key on, we'll
start warning and that'll show up as an UNSTABLE build. These warnings
are more for test maintainers than for public consumption so don't
translate them to NEEDS_INSPECTION because that will block updates on
gated components.

jobs/bodhi-trigger: respect `test-patterns: skip`

If a test pattern is marked as `skip`, then respect that. Don't force
`basic` to run in that case. The logic there was meant as a way to force
`basic` to always run *if* some other tests were also selected, not to
counteract `skip`.

Fixes 6c1f869 ("bodhi-testing.yaml: support specifying tests to run").

jobs/test-override: run `cosa fetch` with `--with-cosa-overrides`

Otherwise, if the override we're testing is pulling in new
dependencies, we won't know to fetch them. This came up in
https://bodhi.fedoraproject.org/updates/FEDORA-2024-969f546f80 which is
adding a dep on `composefs`.

This also matches CI in the coreos/fedora-coreos-config repo.

(Aside: we've talked in the past about getting notified when new deps
are added and the previous behaviour coincidentally did that, but this
doesn't seem like the right place for it; we shouldn't block packages
from entering Fedora on that account. That check probably belongs in
`bump-lockfile` instead).

jobs/bodhi-trigger: send failure notifications to Matrix

This job is still relatively new. Push out failure notifications to
Matrix to increase visibility and make it easier to collaborate on
investigating issues.

docs/hacking: Add matrix-webhook secret and use it in jobs/bodhi-trigger

jobs/bodhi-trigger: Add a link to blueoecean in the matrix message

hacking: fix matrix webhook env var

See https://github.com/coreos/coreos-ci/pull/55#discussion_r1633248050

Jobs/bodhi-trigger update matrix message formatting

Also include a link to the bodhi advisory
Also use propper markdown formatting as
 https://pagure.io/fedora-infrastructure/issue/12057 landed

Fixes #67

jobs/bodhi-trigger: also report test successes

Sometimes, Bodhi testing is broken for a few days if rawhide is in a
bad shape. Let's also have the test report successes so that it's easier
to tell when upstream testing have been fixed. It makes it more obvious
also when e.g. just a single run failed in a series of otherwise passing
runs.

jobs/bodhi-trigger: print raw JSON instead of parsed object

That makes it easier to copy/paste if we want to replay a job (currently
even replaying a fedmsg-triggered job will still ask you to input the
`CI_MESSAGE`).

bodhi-testing: add kexec-tools and makedumpfile

We need to also tests theses, as we need them to support kdump.
This would have caught https://github.com/coreos/fedora-coreos-tracker/issues/1813

jobs/test-override: respect env vars for stream

This matches what we do in the pipeline.

Motivated by wanting to get to the bottom of
https://github.com/coreos/fedora-coreos-tracker/issues/1796, which
happens often in the Bodhi tests.

upstream-ci: test upstream kdump builds against FCOS

In an effort to reduce kdump issues, let's run our tests against their
upstream repo, to catch problems earlier.

See https://github.com/rhkdump/kdump-utils/pull/62
Also: https://issues.redhat.com/browse/RHEL-70438

vars/kola: deprecate --basic-qemu-scenarios

The argument for --basic-qemu-scenarios has been deprecated.
Depending on the version of cosa, the tests that would be skipped by these args are
now formal kola tests. And to skip them you can use --denylist-test commands.

skipSecureBoot now translates to  --denylist-test *.uefi-secure

vars/kola: Add support for skipping a list of kola tags

Introduce a kola parameter to allow skipping a list of kola tags.
This is particularly useful when adding new streams to the pipeline.

xref: https://github.com/coreos/fedora-coreos-pipeline/pull/1008

Reference: https://github.com/coreos/fedora-coreos-pipeline/issues/1002

vars/checkoutToDir: fix checkout step syntax

The extensions object's syntax here is incorrect:

> WARNING: Unknown parameter(s) found for class type 'hudson.plugins.git.extensions.impl.RelativeTargetDirectory': recursiveSubmodules

The `recursiveSubmodules` option is part of another extension type we
need to add to the list.