Block Testing Ready+ challenges whose repo does not back up the status

Anyone can set the README status, and several challenges imported from
Nextcloud claimed Testing Ready with an untouched description.yml. For
every challenge claiming Testing Ready or beyond, now also read the repo
tree and description.yml, and block it if description.yml is missing or
still the template, its category disagrees with the README, the flag is
not h4tum{...}, a dynamic flag reaches no container, a deploy_dir or its
Dockerfile is missing, or there is no solve.py.

Blocked challenges keep their claimed status in the counts but are left
out of "Ready for infra testing" and challs.txt, and listed with the
reason in the terminal and the HedgeDoc note. Adds a PyYAML dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-20 19:23:46 +02:00
co-authored by Claude Opus 5.5
parent ec2f444ef3
commit 4c6cb21cb8
2 changed files with 187 additions and 26 deletions
+39 -9
View File
@@ -2,7 +2,12 @@
Reads the `README.md` of every repo in the GitLab challenges group (REST API, no
clone) and shows which categories have challenges in which state, at which
difficulty — in the terminal and as a HedgeDoc note.
difficulty — in the terminal and as a HedgeDoc note. A status of Testing Ready
or beyond is only believed if the repo backs it up (see
[Checking the repo](#checking-the-repo)).
Needs `requests` and `PyYAML` (`python-requests python-yaml` on Arch,
`python3-requests python3-yaml` on Debian).
## Setup
@@ -23,7 +28,7 @@ $EDITOR ctf_status.env # GitLab group URL + token, HedgeDoc note
The token is a fine-grained personal access token on the challenges group with
**Project: Read** (to list the repos) and **Repository: Read** (to download the
READMEs). A plain `.env` with `GL_PAT=` / `GITLAB_URL=` is read as well.
READMEs and `description.yml`, and to list the repo's files). A plain `.env` with `GL_PAT=` / `GITLAB_URL=` is read as well.
## Keeping a HedgeDoc note up to date
@@ -44,7 +49,7 @@ anonymous. HedgeDoc 1.x has no REST API for editing a note, so the script talks
editor's own engine.io/socket.io + OT protocol over long-polling — it reads the
current text, sends one "delete everything, insert this" operation, waits for the
server's `ack`, and reads the note back to confirm. Anyone with the note open sees
the change live. Still `requests`-only, no new dependencies.
the change live. Plain `requests`, no websocket library.
`--hedgedoc <instance>` (POST a *new* note) is unrelated and only works where
anonymous note creation is enabled — on `hedgedoc.cit.tum.de` it is not.
@@ -52,6 +57,7 @@ anonymous note creation is enabled — on `hedgedoc.cit.tum.de` it is not.
## challs.txt for the deployment repo
`--challs` commits the clone URL of every challenge at **Testing Ready or beyond**
that is not blocked (see [Checking the repo](#checking-the-repo))
to `CHALLS_PATH` (default `2026/challs.txt`) in the deployment repo
`CHALLS_REPO_URL`, one per line, the same format as last year's:
@@ -122,14 +128,16 @@ one-line error and a non-zero exit.
1. **✅ Ready for infra testing** — first thing in the note: every challenge at
Testing Ready, Tested or Ready, linked to its repo, so infra knows what it
can deploy and test against right now.
2. **Status by category** — how many challenges each category has per stage
can deploy and test against right now. Blocked ones are not in it.
2. **⛔ Claims Testing Ready+, but the repo disagrees** — only shown when there
are any: what the README claims, and why the repo says otherwise.
3. **Status by category** — how many challenges each category has per stage
(⚪ Planning → 🟠 PoC Draft → 🟡 Testing Ready → 🔵 Tested → 🟢 Ready).
3. **Category × difficulty** — one dot per challenge, coloured by its status, so
gaps and stragglers show up at a glance.
4. **Challenges** — one table per category, linked to the repo, with event,
4. **Category × difficulty** — one dot per challenge, coloured by its status (⛔
when blocked), so gaps and stragglers show up at a glance.
5. **Challenges** — one table per category, linked to the repo, with event,
author, tester and the repo's last activity.
5. **Needs attention** — see below.
6. **Needs attention** — see below.
## Reading the README
@@ -156,6 +164,28 @@ People fill it in loosely, so values are normalised:
is shown next to the challenge. Categories outside the usual list (e.g. `ai`)
get their own row.
## Checking the repo
Anyone can write `Status: Testing Ready`, so for every challenge whose README
claims Testing Ready, Tested or Ready the repo itself is looked at too. Each of
these makes it **blocked** (⛔): it keeps its claimed status in the counts, but
it is left out of *Ready for infra testing* and `challs.txt`, and the reason is
shown in the note.
- no `description.yml` at the top level, or one that does not parse
- `description.yml` still has anything from the template it was copied from
(`netcat-check-multi-container`): its id, name, author `jh`, its flag, or the
`cont1`/`cont2` / `container-1`/`container-2` containers
- its `category` is not one of the README's categories
- the flag is not `h4tum{...}`
- `dynamic: true`, but no container has a `flag:` block to receive it
- a container's `deploy_dir` does not exist or has no `Dockerfile`
- no `solve.py` anywhere in the repo
These are cheap signs that a challenge was never finished, not proof that it
works: a repo can pass every check and still not build or solve. That is what
actually testing it is for.
## What it flags
- challenge at Testing Ready or beyond with no tester (`[Tester Name]`, `??`,