tudor/school_compare
1
0 Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(seo): keep staging out of the search index #111

Merged
tudor merged 2 commits from fix/staging-noindex into main 2026-08-20 22:39:38 +00:00
Conversation 1 Commits 2 Files Changed 3
+147
2 Commits
Author SHA1 Message Date
TudorandClaude Opus 5 b34511e459 chore: record the branch cleanup manifest
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m3s
Details
PR Checks / Backend Smoke (pull_request) Successful in 7s
Details
PR Checks / Build Backend (no push) (pull_request) Successful in 16s
Details
PR Checks / Build Frontend (no push) (pull_request) Successful in 45s
Details
PR Checks / Build Pipeline (no push) (pull_request) Successful in 10s
Details
PR Checks / AI Code Review (Claude) (pull_request) Successful in 2m30s
Details 
79 remote branches deleted: 77 fully merged into main, plus
feat/seo-crawl-hygiene and feat/england-only-corpus, whose content is
preserved on feat/seo-crawl-hygiene-main (PR #110).

Each line carries the SHA, so any branch can be restored with
  git push origin <sha>:refs/heads/<name>

The 14 branches left standing all carry content that differs from main and
none of them is mine to judge abandoned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
 2026-08-20 23:23:30 +01:00
TudorandClaude Opus 5 1fc1e07d21 fix(seo): keep staging out of the search index
PR Checks / Frontend Typecheck + Tests (pull_request) Successful in 1m3s
Details
PR Checks / Backend Smoke (pull_request) Successful in 7s
Details
PR Checks / Build Backend (no push) (pull_request) Successful in 11s
Details
PR Checks / Build Frontend (no push) (pull_request) Canceled after 13s
Details
PR Checks / Build Pipeline (no push) (pull_request) Canceled after 0s
Details
PR Checks / AI Code Review (Claude) (pull_request) Canceled after 0s
Details 
Staging serves the same image as production off stx., with robots.txt saying
Allow: / and no noindex — a fully crawlable duplicate of the site. Nothing
appears indexed today, most likely because the pages canonicalise across to
production, but that is a side effect rather than a control.

X-Robots-Tag, not a robots.txt Disallow. Disallow blocks crawling, which is
not the same as blocking indexing: a disallowed URL can still be indexed from
external links, and blocking the crawl means Google never fetches the page and
so never sees a noindex at all. Staging stays crawlable and answers noindex.

Matched on the staging host explicitly rather than 'any host that is not
production'. The inverted form would cover future environments automatically,
but its failure mode is deindexing production if the Host header ever arrives
rewritten by a proxy — which cannot be verified from here. This form's failure
mode is a new environment being indexable until someone adds it, which is
recoverable. Any new non-production hostname must be added.

The journeys only ever run against staging (deploy.yml passes
STAGING_BASE_URL; promote.yml smoke-polls production without Playwright), so
asserting the header there is safe. The two assertions live in one test
because the halves only work together.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
 2026-08-20 23:21:58 +01:00