Block Testing Ready+ challenges whose repo does not back up the status
Anyone can set the README status, and several challenges imported from
Nextcloud claimed Testing Ready with an untouched description.yml. For
every challenge claiming Testing Ready or beyond, now also read the repo
tree and description.yml, and block it if description.yml is missing or
still the template, its category disagrees with the README, the flag is
not h4tum{...}, a dynamic flag reaches no container, a deploy_dir or its
Dockerfile is missing, or there is no solve.py.
Blocked challenges keep their claimed status in the counts but are left
out of "Ready for infra testing" and challs.txt, and listed with the
reason in the terminal and the HedgeDoc note. Adds a PyYAML dependency.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -2,7 +2,12 @@
|
||||
|
||||
Reads the `README.md` of every repo in the GitLab challenges group (REST API, no
|
||||
clone) and shows which categories have challenges in which state, at which
|
||||
difficulty — in the terminal and as a HedgeDoc note.
|
||||
difficulty — in the terminal and as a HedgeDoc note. A status of Testing Ready
|
||||
or beyond is only believed if the repo backs it up (see
|
||||
[Checking the repo](#checking-the-repo)).
|
||||
|
||||
Needs `requests` and `PyYAML` (`python-requests python-yaml` on Arch,
|
||||
`python3-requests python3-yaml` on Debian).
|
||||
|
||||
## Setup
|
||||
|
||||
@@ -23,7 +28,7 @@ $EDITOR ctf_status.env # GitLab group URL + token, HedgeDoc note
|
||||
|
||||
The token is a fine-grained personal access token on the challenges group with
|
||||
**Project: Read** (to list the repos) and **Repository: Read** (to download the
|
||||
READMEs). A plain `.env` with `GL_PAT=` / `GITLAB_URL=` is read as well.
|
||||
READMEs and `description.yml`, and to list the repo's files). A plain `.env` with `GL_PAT=` / `GITLAB_URL=` is read as well.
|
||||
|
||||
## Keeping a HedgeDoc note up to date
|
||||
|
||||
@@ -44,7 +49,7 @@ anonymous. HedgeDoc 1.x has no REST API for editing a note, so the script talks
|
||||
editor's own engine.io/socket.io + OT protocol over long-polling — it reads the
|
||||
current text, sends one "delete everything, insert this" operation, waits for the
|
||||
server's `ack`, and reads the note back to confirm. Anyone with the note open sees
|
||||
the change live. Still `requests`-only, no new dependencies.
|
||||
the change live. Plain `requests`, no websocket library.
|
||||
|
||||
`--hedgedoc <instance>` (POST a *new* note) is unrelated and only works where
|
||||
anonymous note creation is enabled — on `hedgedoc.cit.tum.de` it is not.
|
||||
@@ -52,6 +57,7 @@ anonymous note creation is enabled — on `hedgedoc.cit.tum.de` it is not.
|
||||
## challs.txt for the deployment repo
|
||||
|
||||
`--challs` commits the clone URL of every challenge at **Testing Ready or beyond**
|
||||
that is not blocked (see [Checking the repo](#checking-the-repo))
|
||||
to `CHALLS_PATH` (default `2026/challs.txt`) in the deployment repo
|
||||
`CHALLS_REPO_URL`, one per line, the same format as last year's:
|
||||
|
||||
@@ -122,14 +128,16 @@ one-line error and a non-zero exit.
|
||||
|
||||
1. **✅ Ready for infra testing** — first thing in the note: every challenge at
|
||||
Testing Ready, Tested or Ready, linked to its repo, so infra knows what it
|
||||
can deploy and test against right now.
|
||||
2. **Status by category** — how many challenges each category has per stage
|
||||
can deploy and test against right now. Blocked ones are not in it.
|
||||
2. **⛔ Claims Testing Ready+, but the repo disagrees** — only shown when there
|
||||
are any: what the README claims, and why the repo says otherwise.
|
||||
3. **Status by category** — how many challenges each category has per stage
|
||||
(⚪ Planning → 🟠 PoC Draft → 🟡 Testing Ready → 🔵 Tested → 🟢 Ready).
|
||||
3. **Category × difficulty** — one dot per challenge, coloured by its status, so
|
||||
gaps and stragglers show up at a glance.
|
||||
4. **Challenges** — one table per category, linked to the repo, with event,
|
||||
4. **Category × difficulty** — one dot per challenge, coloured by its status (⛔
|
||||
when blocked), so gaps and stragglers show up at a glance.
|
||||
5. **Challenges** — one table per category, linked to the repo, with event,
|
||||
author, tester and the repo's last activity.
|
||||
5. **Needs attention** — see below.
|
||||
6. **Needs attention** — see below.
|
||||
|
||||
## Reading the README
|
||||
|
||||
@@ -156,6 +164,28 @@ People fill it in loosely, so values are normalised:
|
||||
is shown next to the challenge. Categories outside the usual list (e.g. `ai`)
|
||||
get their own row.
|
||||
|
||||
## Checking the repo
|
||||
|
||||
Anyone can write `Status: Testing Ready`, so for every challenge whose README
|
||||
claims Testing Ready, Tested or Ready the repo itself is looked at too. Each of
|
||||
these makes it **blocked** (⛔): it keeps its claimed status in the counts, but
|
||||
it is left out of *Ready for infra testing* and `challs.txt`, and the reason is
|
||||
shown in the note.
|
||||
|
||||
- no `description.yml` at the top level, or one that does not parse
|
||||
- `description.yml` still has anything from the template it was copied from
|
||||
(`netcat-check-multi-container`): its id, name, author `jh`, its flag, or the
|
||||
`cont1`/`cont2` / `container-1`/`container-2` containers
|
||||
- its `category` is not one of the README's categories
|
||||
- the flag is not `h4tum{...}`
|
||||
- `dynamic: true`, but no container has a `flag:` block to receive it
|
||||
- a container's `deploy_dir` does not exist or has no `Dockerfile`
|
||||
- no `solve.py` anywhere in the repo
|
||||
|
||||
These are cheap signs that a challenge was never finished, not proof that it
|
||||
works: a repo can pass every check and still not build or solve. That is what
|
||||
actually testing it is for.
|
||||
|
||||
## What it flags
|
||||
|
||||
- challenge at Testing Ready or beyond with no tester (`[Tester Name]`, `??`,
|
||||
|
||||
Reference in New Issue
Block a user