The staging gate was red on three journeys. All three were faults in the tests — the site was behaving correctly in every case. No product change here.
1 & 2. The homepage canonical has no trailing slash, and that is correct
Next normalises canonical URLs against trailingSlash: false, so the root ships without a slash while every other route keeps its path. Both forms address the same document.
The tell was in the results: /rankings and /admissions passed throughout and only / failed, because only the root has an empty path to normalise. I hardcoded the slash.
Now compared with trailing slashes stripped from both sides — which form Next emits is its business, not something worth pinning a test to.
3. The blanket Disallow: / belongs to the AI crawlers, not to *
The assertion matched Disallow: /anywhere in the file. Cloudflare injects managed blocks for nine AI crawlers — ClaudeBot, GPTBot, Amazonbot, CCBot and others — each carrying a deliberate blanket disallow. None of them is Googlebot.
It now parses the file into user-agent groups and checks only *. That is also what the test was always trying to say: Google may crawl the page, so it can see the X-Robots-Tag: noindex. The old regex could not express that.
Both are the same mistake as the doubled brand: asserting a naive string rather than the semantics, and asserting against what the code assembles rather than what the page renders. The robots fix in particular is now a better test than the original — it expresses the actual requirement instead of a proxy for it.
The staging gate was red on three journeys. **All three were faults in the tests** — the site was behaving correctly in every case. No product change here.
## 1 & 2. The homepage canonical has no trailing slash, and that is correct
```
Expected: "https://www.schoolcompare.co.uk/"
Received: "https://www.schoolcompare.co.uk"
```
Next normalises canonical URLs against `trailingSlash: false`, so the root ships without a slash while every other route keeps its path. Both forms address the same document.
The tell was in the results: `/rankings` and `/admissions` passed throughout and only `/` failed, because only the root has an empty path to normalise. I hardcoded the slash.
Now compared with trailing slashes stripped from both sides — which form Next emits is its business, not something worth pinning a test to.
## 3. The blanket `Disallow: /` belongs to the AI crawlers, not to `*`
The assertion matched `Disallow: /` **anywhere in the file**. Cloudflare injects managed blocks for nine AI crawlers — ClaudeBot, GPTBot, Amazonbot, CCBot and others — each carrying a deliberate blanket disallow. None of them is Googlebot.
The `*` group actually reads:
```
User-Agent: *
Allow: /
Disallow: /api/
Disallow: /_next/
```
It now parses the file into user-agent groups and checks only `*`. That is also what the test was always trying to say: Google may crawl the page, so it can see the `X-Robots-Tag: noindex`. The old regex could not express that.
## Verified against live staging
Rather than only against the suite:
| Assertion | Result |
|---|---|
| `/` canonical | PASS — `https://www.schoolcompare.co.uk` |
| `/rankings` canonical | PASS |
| `/admissions` canonical | PASS |
| `/?search=primary` canonical | PASS — collapses to root |
| `*` group blocks everything | PASS — False |
| (sanity) ClaudeBot blocked | True — the parser distinguishes groups |
Backend 110 · frontend 258 · `tsc --noEmit` clean · 85 journeys parse.
## The pattern
Both are the same mistake as the doubled brand: asserting a naive string rather than the semantics, and asserting against what the code assembles rather than what the page renders. The robots fix in particular is now a better test than the original — it expresses the actual requirement instead of a proxy for it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
The staging gate was red on three journeys. All three were faults in the
tests; the site was behaving correctly in each case.
Next normalises canonical URLs against trailingSlash:false, so the homepage
ships "https://www.schoolcompare.co.uk" with no slash while every other route
keeps its path. Both address the same document. The test hardcoded the slash
and so failed only on the root — /rankings and /admissions passed throughout,
which is what made it look like a homepage bug rather than a test bug.
Compared with trailing slashes stripped from both sides.
The robots.txt assertion matched "Disallow: /" anywhere in the file and
tripped over the AI-crawler groups Cloudflare injects — ClaudeBot, GPTBot,
Amazonbot and six others all carry a blanket disallow, deliberately, and none
of them is Googlebot. It now parses the file into user-agent groups and checks
only the "*" group, which is also the thing the test was always trying to say:
Google may crawl the page, so it can see the noindex header.
Both were the same mistake as the doubled brand: asserting a naive string
rather than the semantics, and asserting against what the code assembles
rather than what the page renders.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
This PR loosens overly strict e2e assertions for canonical-URL and robots.txt checks, replacing exact string equality with helper functions that tolerate benign formatting differences (trailing slash, multi-group robots.txt). It only touches test code, so there's no production risk, but the new helpers introduce their own minor correctness gaps.
🟡 Minor
e2e/tests/journeys.spec.ts: sameUrl() strips trailing slashes for every route, not just the root. Per the comment, only the homepage is expected to vary (trailingSlash: false); applying the same tolerance to all other routes means a regression that accidentally appends a trailing slash to e.g. /admissions's canonical would now silently pass instead of failing.
e2e/tests/journeys.spec.ts: blocksEverything() tracks only a single current user-agent, overwritten on each User-agent line. A robots.txt group with multiple User-agent lines sharing one Disallow (e.g. 'User-agent: ' followed by 'User-agent: Googlebot' then 'Disallow: /') would only attribute the Disallow to the last-seen agent, so a blanket disallow on '' in such a grouped block would go undetected (false negative).
## 🤖 AI Code Review (Claude Code)
This PR loosens overly strict e2e assertions for canonical-URL and robots.txt checks, replacing exact string equality with helper functions that tolerate benign formatting differences (trailing slash, multi-group robots.txt). It only touches test code, so there's no production risk, but the new helpers introduce their own minor correctness gaps.
### 🟡 Minor
- **e2e/tests/journeys.spec.ts**: sameUrl() strips trailing slashes for every route, not just the root. Per the comment, only the homepage is expected to vary (trailingSlash: false); applying the same tolerance to all other routes means a regression that accidentally appends a trailing slash to e.g. /admissions's canonical would now silently pass instead of failing.
- **e2e/tests/journeys.spec.ts**: blocksEverything() tracks only a single `current` user-agent, overwritten on each User-agent line. A robots.txt group with multiple User-agent lines sharing one Disallow (e.g. 'User-agent: *' followed by 'User-agent: Googlebot' then 'Disallow: /') would only attribute the Disallow to the last-seen agent, so a blanket disallow on '*' in such a grouped block would go undetected (false negative).
tudor
merged commit 4a9a5c734b into main2026-08-21 23:07:58 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The staging gate was red on three journeys. All three were faults in the tests — the site was behaving correctly in every case. No product change here.
1 & 2. The homepage canonical has no trailing slash, and that is correct
Next normalises canonical URLs against
trailingSlash: false, so the root ships without a slash while every other route keeps its path. Both forms address the same document.The tell was in the results:
/rankingsand/admissionspassed throughout and only/failed, because only the root has an empty path to normalise. I hardcoded the slash.Now compared with trailing slashes stripped from both sides — which form Next emits is its business, not something worth pinning a test to.
3. The blanket
Disallow: /belongs to the AI crawlers, not to*The assertion matched
Disallow: /anywhere in the file. Cloudflare injects managed blocks for nine AI crawlers — ClaudeBot, GPTBot, Amazonbot, CCBot and others — each carrying a deliberate blanket disallow. None of them is Googlebot.The
*group actually reads:It now parses the file into user-agent groups and checks only
*. That is also what the test was always trying to say: Google may crawl the page, so it can see theX-Robots-Tag: noindex. The old regex could not express that.Verified against live staging
Rather than only against the suite:
/canonicalhttps://www.schoolcompare.co.uk/rankingscanonical/admissionscanonical/?search=primarycanonical*group blocks everythingBackend 110 · frontend 258 ·
tsc --noEmitclean · 85 journeys parse.The pattern
Both are the same mistake as the doubled brand: asserting a naive string rather than the semantics, and asserting against what the code assembles rather than what the page renders. The robots fix in particular is now a better test than the original — it expresses the actual requirement instead of a proxy for it.
🤖 Generated with Claude Code
https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
🤖 AI Code Review (Claude Code)
This PR loosens overly strict e2e assertions for canonical-URL and robots.txt checks, replacing exact string equality with helper functions that tolerate benign formatting differences (trailing slash, multi-group robots.txt). It only touches test code, so there's no production risk, but the new helpers introduce their own minor correctness gaps.
🟡 Minor
currentuser-agent, overwritten on each User-agent line. A robots.txt group with multiple User-agent lines sharing one Disallow (e.g. 'User-agent: ' followed by 'User-agent: Googlebot' then 'Disallow: /') would only attribute the Disallow to the last-seen agent, so a blanket disallow on '' in such a grouped block would go undetected (false negative).