Files
school_compare/docs/superpowers/plans/2026-08-26-school-autosuggest.md
T

1359 lines
48 KiB
Markdown
Raw Normal View History

# School Autosuggest Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Suggest schools by name as someone types in the site's main search box, so a parent who knows the school they want reaches it in one step.
**Architecture:** A dedicated `/api/suggest` endpoint answers from Typesense alone, touching no pandas. The frontend adds an ARIA combobox to the existing `FilterBar` omni-input, behind the `school_autosuggest` flag. First, the rate limiter is fixed to key on the real caller rather than the Next container — without that, autosuggest 429s the site.
**Tech Stack:** FastAPI, slowapi, Typesense, Next.js 15 App Router, React, Playwright, pytest, Jest.
**Spec:** `docs/superpowers/specs/2026-08-26-school-autosuggest-design.md`
## Global Constraints
- **The rate-limit fix is not flagged.** It is a correctness fix that applies whether or not autosuggest is on.
- **`/api/suggest` is not flagged either.** Only the UI is. A live endpoint with the UI dark is deliberate — it lets the endpoint be smoke-tested before the feature is switched on.
- **The keystroke path never returns an error for ordinary input.** Short query, no matches, Typesense down — all `200` with `{"suggestions": []}`.
- **No DataFrame fallback in `/api/suggest`.** The 25,000-row substring scan `/api/schools` falls back to is exactly the cost this endpoint exists to avoid.
- **`Enter` with no active option must still submit the free-text search**, exactly as today.
- **The client fetch must not use `cache: "no-store"`** — it would discard the browser cache and the ETag 304s the existing middleware already provides.
- **Every flag defaults to `False`**, declared in `backend/flags.py` with a `name`/`description`/`added` triple.
- Backend tests run via: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests -q`
- Never push to `main`. Branch, PR, let checks pass.
- Do not start a local server to test the application; it does not work.
## File Structure
| File | Responsibility |
|---|---|
| `backend/app.py` (modify) | `client_key`, the `Limiter` construction, `GET /api/suggest`, one `CACHE_RULES` row. |
| `backend/data_loader.py` (modify) | `suggest_schools_typesense` — returns documents, not URNs. |
| `backend/flags.py` (modify) | Declare `school_autosuggest`. |
| `backend/tests/test_rate_limit_key.py` (new) | Keying precedence and bucket separation. |
| `backend/tests/test_suggest.py` (new) | Endpoint behaviour, including that it never touches the DataFrame. |
| `nextjs-app/lib/suggest.ts` (new) | `fetchSuggestions(q, signal)` — one typed fetch. |
| `nextjs-app/hooks/useSchoolSuggest.ts` (new) | Debounce, abort, stale-response rejection, open/active state. |
| `nextjs-app/components/SuggestList.tsx` (new) | Presentational list + ARIA. No fetching. |
| `nextjs-app/components/SuggestList.module.css` (new) | Dropdown styling. |
| `nextjs-app/components/FilterBar.tsx` (modify) | Wire hook + list to the existing input and form. |
| `nextjs-app/components/HomeView.tsx` (modify) | Pass `autosuggest` through to both `FilterBar` instances. |
| `nextjs-app/app/page.tsx` (modify) | Read the flag server-side, pass to both `HomeView` render sites. |
| `e2e/tests/journeys.spec.ts` (modify) | Flag-gated journeys. |
---
### Task 1: Key the rate limiter on the real caller
**Files:**
- Modify: `backend/app.py`
- Create: `backend/tests/test_rate_limit_key.py`
**Interfaces:**
- Produces: `backend.app.client_key(request: Request) -> str`
> Measured on staging before this change: 70 concurrent requests to
> `/api/schools` returned exactly 60 × 200 and 10 × 429. One machine consumed
> the whole site's budget, because `get_remote_address` returns the Next
> container's IP for every browser user.
- [ ] **Step 1: Write the failing test**
Create `backend/tests/test_rate_limit_key.py`:
```python
"""The rate-limit bucket must be the caller, not the proxy in front of them.
`get_remote_address` reads request.client.host. In staging and production the
backend has no published ports and its only caller is the Next proxy, so that
host is the Next container — one bucket for every browser user on the site.
Measured before this fix: 70 concurrent requests, 60 served and 10 refused.
"""
from starlette.datastructures import Headers
from backend.app import client_key
class _Req:
"""Enough of a Request for the key function: headers and a client host."""
def __init__(self, headers: dict, host: str = "10.0.0.9"):
self.headers = Headers(headers)
self.client = type("C", (), {"host": host})()
self.scope = {"type": "http", "client": (host, 0),
"headers": [(k.lower().encode(), v.encode())
for k, v in headers.items()]}
def test_cloudflare_header_wins():
# Cloudflare sets CF-Connecting-IP and overwrites any client-supplied
# value, so it is trustworthy in a way a parsed XFF chain is not.
assert client_key(_Req({"cf-connecting-ip": "203.0.113.7"})) == "203.0.113.7"
def test_forwarded_for_is_the_fallback_and_takes_the_first_entry():
# Left-most is the original client; everything after it is proxies.
assert client_key(
_Req({"x-forwarded-for": "203.0.113.7, 10.0.0.2"})) == "203.0.113.7"
def test_remote_address_is_the_last_resort():
assert client_key(_Req({}, host="10.0.0.9")) == "10.0.0.9"
def test_cloudflare_header_beats_forwarded_for():
key = client_key(_Req({"cf-connecting-ip": "203.0.113.7",
"x-forwarded-for": "198.51.100.1"}))
assert key == "203.0.113.7"
def test_two_callers_get_two_buckets():
# The whole point: one user exhausting their limit must not refuse another.
a = client_key(_Req({"cf-connecting-ip": "203.0.113.7"}))
b = client_key(_Req({"cf-connecting-ip": "203.0.113.8"}))
assert a != b
def test_whitespace_is_stripped():
# "a, b" split on comma leaves a leading space on every entry but the
# first; an unstripped key silently creates a second bucket per client.
assert client_key(_Req({"x-forwarded-for": " 203.0.113.7 ,10.0.0.2"})) \
== "203.0.113.7"
```
- [ ] **Step 2: Run it to verify it fails**
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests/test_rate_limit_key.py -q`
Expected: FAIL — `ImportError: cannot import name 'client_key' from 'backend.app'`
- [ ] **Step 3: Implement the key function**
In `backend/app.py`, replace:
```python
# Rate limiter
limiter = Limiter(key_func=get_remote_address)
```
with:
```python
def client_key(request: Request) -> str:
"""The rate-limit bucket: the real caller, not the proxy in front of them.
`get_remote_address` reads request.client.host. In staging and production
the backend has no published ports and sits on the internal network, so its
only caller is the Next proxy — meaning every browser user on the site
shared one bucket. Measured before this fix: 70 concurrent requests to
/api/schools returned 60 OK and 10 refused.
CF-Connecting-IP first, because Cloudflare (in front of both environments)
sets it on every origin request and *overwrites* any client-supplied value,
which a parsed X-Forwarded-For chain does not guarantee. The XFF fallback is
forgeable, but only by a caller already inside the Docker network, which is
the one place nothing untrusted can reach.
"""
cf = request.headers.get("cf-connecting-ip")
if cf:
return cf.strip()
xff = request.headers.get("x-forwarded-for")
if xff:
return xff.split(",")[0].strip()
return get_remote_address(request)
# Rate limiter. No in-app global ceiling: slowapi's default_limits and
# application_limits are both keyed by key_func (so per-client, not global)
# and the latter only applies with SlowAPIMiddleware installed, which this app
# does not use. A global cap belongs at Cloudflare, which is already in the
# path. See the spec's §1 for why that is deliberate.
limiter = Limiter(key_func=client_key)
```
- [ ] **Step 4: Run the tests to verify they pass**
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests/test_rate_limit_key.py -q`
Expected: PASS, 6 tests.
- [ ] **Step 5: Run the full backend suite**
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests -q`
Expected: all pass.
- [ ] **Step 6: Commit**
```bash
git add backend/app.py backend/tests/test_rate_limit_key.py
git commit -m "fix(api): rate-limit per caller, not per proxy"
```
---
### Task 2: `suggest_schools_typesense`
**Files:**
- Modify: `backend/data_loader.py`
- Create: `backend/tests/test_suggest.py`
**Interfaces:**
- Produces: `backend.data_loader.suggest_schools_typesense(query: str, limit: int = 8) -> list[dict]`, each dict carrying `urn` (int), `school_name`, `local_authority`, `postcode`, `phase`, `school_type` (all str, `""` when the document omits an optional field).
> The existing `search_schools_typesense` returns URNs only, which forces the
> caller to hydrate from the DataFrame. Suggestions need the fields Typesense
> already holds — see the schema in `pipeline/scripts/sync_typesense.py`.
- [ ] **Step 1: Write the failing test**
Create `backend/tests/test_suggest.py`:
```python
"""Tests for school autosuggest (spec 2026-08-26)."""
import pytest
from backend import data_loader
class _FakeDocs:
def __init__(self, hits, explode=False):
self._hits = hits
self._explode = explode
self.last_params = None
def search(self, params):
self.last_params = params
if self._explode:
raise RuntimeError("typesense is down")
return {"hits": [{"document": d} for d in self._hits]}
class _FakeClient:
def __init__(self, hits, explode=False):
self.docs = _FakeDocs(hits, explode)
self.collections = {"schools": type("C", (), {"documents": self.docs})()}
_HIT = {
"urn": 100010, "school_name": "Brecknock Primary School",
"local_authority": "Camden", "postcode": "NW1 1AA",
"phase": "Primary", "school_type": "Community school",
}
def _use(monkeypatch, client):
monkeypatch.setattr(data_loader, "_get_typesense_client", lambda: client)
def test_returns_the_fields_a_suggestion_needs(monkeypatch):
# Local authority is not decoration: there are many schools called
# "St Mary's", and a list without it cannot be chosen between.
_use(monkeypatch, _FakeClient([_HIT]))
out = data_loader.suggest_schools_typesense("breck")
assert out == [{
"urn": 100010, "school_name": "Brecknock Primary School",
"local_authority": "Camden", "postcode": "NW1 1AA",
"phase": "Primary", "school_type": "Community school",
}]
def test_a_missing_optional_field_becomes_an_empty_string(monkeypatch):
# phase and school_type are optional in the Typesense schema. A missing
# key must not KeyError in the keystroke path.
_use(monkeypatch, _FakeClient([{"urn": 1, "school_name": "X",
"local_authority": "Y", "postcode": "Z"}]))
out = data_loader.suggest_schools_typesense("x")
assert out[0]["phase"] == "" and out[0]["school_type"] == ""
def test_typesense_unavailable_gives_no_suggestions_rather_than_raising(monkeypatch):
_use(monkeypatch, None)
assert data_loader.suggest_schools_typesense("anything") == []
def test_a_typesense_error_gives_no_suggestions_rather_than_raising(monkeypatch):
_use(monkeypatch, _FakeClient([], explode=True))
assert data_loader.suggest_schools_typesense("anything") == []
def test_the_limit_is_passed_through_and_clamped(monkeypatch):
client = _FakeClient([])
_use(monkeypatch, client)
data_loader.suggest_schools_typesense("x", limit=500)
assert client.docs.last_params["per_page"] == 20
```
- [ ] **Step 2: Run it to verify it fails**
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests/test_suggest.py -q`
Expected: FAIL — `AttributeError: module 'backend.data_loader' has no attribute 'suggest_schools_typesense'`
- [ ] **Step 3: Implement it**
In `backend/data_loader.py`, directly below `search_schools_typesense`:
```python
# The most a public endpoint will return in one response.
SUGGEST_MAX_LIMIT = 20
# Fields a suggestion row carries, and the default when the document omits an
# optional one. phase and school_type are optional in the Typesense schema.
_SUGGEST_FIELDS = ("school_name", "local_authority", "postcode",
"phase", "school_type")
def suggest_schools_typesense(query: str, limit: int = 8) -> List[dict]:
"""Autosuggest rows straight from Typesense. Never raises.
Returns documents rather than URNs, unlike search_schools_typesense, so the
caller needs no DataFrame. Every field below is already in the index — see
pipeline/scripts/sync_typesense.py — which is what makes this cheap enough
to run per keystroke.
"""
client = _get_typesense_client()
if client is None:
return []
try:
result = client.collections["schools"].documents.search({
"q": query,
"query_by": "school_name,local_authority",
"per_page": max(1, min(limit, SUGGEST_MAX_LIMIT)),
"typo_tokens_threshold": 1,
})
except Exception:
# A dropdown that quietly stops appearing is the right failure here.
return []
rows = []
for hit in result.get("hits", []):
doc = hit.get("document", {})
row = {"urn": int(doc.get("urn", 0))}
row.update({f: str(doc.get(f, "") or "") for f in _SUGGEST_FIELDS})
rows.append(row)
return rows
```
- [ ] **Step 4: Run the tests to verify they pass**
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests/test_suggest.py -q`
Expected: PASS, 5 tests.
- [ ] **Step 5: Commit**
```bash
git add backend/data_loader.py backend/tests/test_suggest.py
git commit -m "feat(suggest): Typesense rows for autosuggest, no DataFrame"
```
---
### Task 3: `GET /api/suggest`
**Files:**
- Modify: `backend/app.py`
- Modify: `backend/tests/test_suggest.py`
**Interfaces:**
- Consumes: `backend.data_loader.suggest_schools_typesense(query, limit) -> list[dict]`
- Produces: `GET /api/suggest?q=<str>&limit=<int>` → `200 {"suggestions": [...]}`
- [ ] **Step 1: Write the failing tests**
Append to `backend/tests/test_suggest.py`:
```python
def _client(monkeypatch, rows, *, blow_up_dataframe=False):
from fastapi.testclient import TestClient
from backend import app as app_module
monkeypatch.setattr(app_module, "suggest_schools_typesense",
lambda q, limit=8: rows)
if blow_up_dataframe:
def _boom():
raise AssertionError("the suggest path must not load the DataFrame")
monkeypatch.setattr(app_module, "load_school_data", _boom)
monkeypatch.setattr(app_module, "load_latest_school_data", _boom)
return TestClient(app_module.app, raise_server_exceptions=False)
def test_the_endpoint_returns_suggestions(monkeypatch):
body = _client(monkeypatch, [_HIT]).get("/api/suggest?q=breck").json()
assert body["suggestions"][0]["school_name"] == "Brecknock Primary School"
def test_the_endpoint_never_touches_the_dataframe(monkeypatch):
"""The whole reason this is not a mode of /api/schools.
That endpoint filters and sorts 25,000 rows of pandas per query, holding
the GIL. Per keystroke, that is the cost this endpoint exists to avoid.
"""
res = _client(monkeypatch, [_HIT], blow_up_dataframe=True).get("/api/suggest?q=breck")
assert res.status_code == 200
assert res.json()["suggestions"]
def test_a_one_character_query_returns_nothing_and_does_not_error(monkeypatch):
# The keystroke path never errors on ordinary input.
res = _client(monkeypatch, [_HIT]).get("/api/suggest?q=b")
assert res.status_code == 200
assert res.json() == {"suggestions": []}
def test_a_blank_query_returns_nothing_and_does_not_error(monkeypatch):
res = _client(monkeypatch, [_HIT]).get("/api/suggest?q=")
assert res.status_code == 200
assert res.json() == {"suggestions": []}
def test_typesense_down_is_an_empty_list_not_a_500(monkeypatch):
res = _client(monkeypatch, []).get("/api/suggest?q=breck")
assert res.status_code == 200
assert res.json() == {"suggestions": []}
def test_the_response_is_cacheable(monkeypatch):
# Prefix queries repeat enormously across users, and school names change
# once a year. Without this the endpoint pays full price every keystroke.
res = _client(monkeypatch, [_HIT]).get("/api/suggest?q=breck")
assert "s-maxage" in res.headers.get("cache-control", "")
assert res.headers.get("etag")
```
- [ ] **Step 2: Run them to verify they fail**
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests/test_suggest.py -q`
Expected: FAIL — the endpoint 404s, so `body["suggestions"]` raises `KeyError`.
- [ ] **Step 3: Import the helper**
In `backend/app.py`, add `suggest_schools_typesense` to the existing
`from .data_loader import (...)` block, alongside `search_schools_typesense`.
- [ ] **Step 4: Add the cache rule**
In `backend/app.py`, add to `CACHE_RULES` — before the `/api/schools` row, so
the longest-prefix match is unaffected:
```python
("/api/suggest", (60, 3600, 86400)),
```
- [ ] **Step 5: Add the endpoint**
In `backend/app.py`, directly above `@app.get("/api/flags")`:
```python
# Two characters. One is not a query — it matches thousands of schools and the
# response is useless, so it is not worth a round trip.
SUGGEST_MIN_QUERY = 2
@app.get("/api/suggest")
@limiter.limit("120/minute")
async def suggest_schools(
request: Request,
q: str = Query("", max_length=100),
limit: int = Query(8, ge=1, le=20),
):
"""School name suggestions, from Typesense alone.
Deliberately not a mode of /api/schools: that path filters and sorts the
full in-memory DataFrame, which is far too expensive to run per keystroke.
Nothing here returns an error for ordinary input. A short query, no
matches, or Typesense being unreachable are all 200 with an empty list —
a dropdown that quietly does not appear is the right failure for a
keystroke path, and there is no DataFrame fallback because the 25,000-row
substring scan is precisely what this endpoint exists to avoid.
120/minute rather than the default 60: a 200 ms debounce makes typing
legitimately bursty.
"""
query = q.strip()
if len(query) < SUGGEST_MIN_QUERY:
return {"suggestions": []}
return {"suggestions": suggest_schools_typesense(query, limit)}
```
- [ ] **Step 6: Run the tests to verify they pass**
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests -q`
Expected: all pass.
- [ ] **Step 7: Commit**
```bash
git add backend/app.py backend/tests/test_suggest.py
git commit -m "feat(suggest): GET /api/suggest, cacheable and DataFrame-free"
```
---
### Task 4: The client and the hook
**Files:**
- Create: `nextjs-app/lib/suggest.ts`
- Create: `nextjs-app/hooks/useSchoolSuggest.ts`
- Create: `nextjs-app/__tests__/hooks/useSchoolSuggest.test.tsx`
**Interfaces:**
- Produces:
- `export interface Suggestion { urn: number; school_name: string; local_authority: string; postcode: string; phase: string; school_type: string }`
- `fetchSuggestions(q: string, signal?: AbortSignal): Promise<Suggestion[]>`
- `useSchoolSuggest(query: string, enabled: boolean): { suggestions: Suggestion[]; open: boolean; activeIndex: number; setActiveIndex: (i: number) => void; close: () => void }`
- [ ] **Step 1: Write `lib/suggest.ts`**
```typescript
/**
* Client for /api/suggest.
*
* No `cache: "no-store"`. The compare modal's search uses it, and copying that
* here would discard both the browser cache and the ETag 304s the backend's
* CacheAndETagMiddleware already provides — on the one endpoint where prefix
* queries repeat most.
*/
export interface Suggestion {
urn: number;
school_name: string;
local_authority: string;
postcode: string;
phase: string;
school_type: string;
}
/** Below this the response is thousands of schools and worth no round trip. */
export const SUGGEST_MIN_QUERY = 2;
const API = process.env.NEXT_PUBLIC_API_URL || '/api';
/** Suggestions for `q`. Never throws: no suggestions is a fine outcome. */
export async function fetchSuggestions(
q: string, signal?: AbortSignal,
): Promise<Suggestion[]> {
if (q.trim().length < SUGGEST_MIN_QUERY) return [];
try {
const res = await fetch(`${API}/suggest?q=${encodeURIComponent(q.trim())}`,
{ signal });
if (!res.ok) return [];
const body = await res.json();
return body.suggestions ?? [];
} catch {
// Includes AbortError, which is the normal path on every keystroke.
return [];
}
}
```
- [ ] **Step 2: Write the failing hook test**
Create `nextjs-app/__tests__/hooks/useSchoolSuggest.test.tsx`:
```tsx
import { renderHook, act, waitFor } from '@testing-library/react';
import { useSchoolSuggest } from '@/hooks/useSchoolSuggest';
const realFetch = global.fetch;
function mockFetch(rows: unknown[], delayMs = 0) {
global.fetch = jest.fn(async (_url: unknown, init?: { signal?: AbortSignal }) => {
if (delayMs) {
await new Promise((resolve, reject) => {
const t = setTimeout(resolve, delayMs);
init?.signal?.addEventListener('abort', () => {
clearTimeout(t);
reject(Object.assign(new Error('aborted'), { name: 'AbortError' }));
});
});
}
return { ok: true, json: async () => ({ suggestions: rows }) };
}) as unknown as typeof fetch;
}
const ROW = {
urn: 1, school_name: 'Brecknock Primary School', local_authority: 'Camden',
postcode: 'NW1 1AA', phase: 'Primary', school_type: 'Community school',
};
describe('useSchoolSuggest', () => {
beforeEach(() => { jest.useFakeTimers(); });
afterEach(() => { jest.useRealTimers(); global.fetch = realFetch; });
it('does not fetch below the minimum query length', () => {
mockFetch([ROW]);
renderHook(() => useSchoolSuggest('b', true));
act(() => { jest.advanceTimersByTime(500); });
expect(global.fetch).not.toHaveBeenCalled();
});
it('does not fetch at all when disabled', () => {
// The flag being off must mean no request, not a hidden dropdown.
mockFetch([ROW]);
renderHook(() => useSchoolSuggest('brecknock', false));
act(() => { jest.advanceTimersByTime(500); });
expect(global.fetch).not.toHaveBeenCalled();
});
it('debounces rather than firing per keystroke', () => {
mockFetch([ROW]);
const { rerender } = renderHook(
({ q }) => useSchoolSuggest(q, true), { initialProps: { q: 'br' } });
rerender({ q: 'bre' });
rerender({ q: 'brec' });
act(() => { jest.advanceTimersByTime(199); });
expect(global.fetch).not.toHaveBeenCalled();
act(() => { jest.advanceTimersByTime(2); });
expect(global.fetch).toHaveBeenCalledTimes(1);
});
it('opens with results once they arrive', async () => {
mockFetch([ROW]);
const { result } = renderHook(() => useSchoolSuggest('brecknock', true));
act(() => { jest.advanceTimersByTime(200); });
await waitFor(() => expect(result.current.suggestions).toHaveLength(1));
expect(result.current.open).toBe(true);
});
it('close() hides the list without clearing the query', async () => {
mockFetch([ROW]);
const { result } = renderHook(() => useSchoolSuggest('brecknock', true));
act(() => { jest.advanceTimersByTime(200); });
await waitFor(() => expect(result.current.open).toBe(true));
act(() => { result.current.close(); });
expect(result.current.open).toBe(false);
});
});
```
- [ ] **Step 3: Run it to verify it fails**
Run: `cd nextjs-app && npx jest __tests__/hooks/useSchoolSuggest.test.tsx`
Expected: FAIL — `Cannot find module '@/hooks/useSchoolSuggest'`
- [ ] **Step 4: Write the hook**
Create `nextjs-app/hooks/useSchoolSuggest.ts`:
```typescript
'use client';
import { useEffect, useRef, useState } from 'react';
import { fetchSuggestions, SUGGEST_MIN_QUERY, type Suggestion } from '@/lib/suggest';
/*
* Long enough that a fast typist does not fire a request per character, short
* enough that the list feels attached to the keyboard.
*/
const DEBOUNCE_MS = 200;
export function useSchoolSuggest(query: string, enabled: boolean) {
const [suggestions, setSuggestions] = useState<Suggestion[]>([]);
const [open, setOpen] = useState(false);
const [activeIndex, setActiveIndex] = useState(-1);
// Set when the user dismisses the list, so a re-render does not reopen it.
const dismissed = useRef('');
useEffect(() => {
const q = query.trim();
if (!enabled || q.length < SUGGEST_MIN_QUERY || dismissed.current === q) {
setSuggestions([]);
setOpen(false);
return;
}
/*
* Abort the superseded request on every keystroke. This is correctness,
* not economy: without it a slow response for "st" can land after the fast
* one for "st marys" and replace a correct list with a stale one.
*/
const controller = new AbortController();
const timer = setTimeout(async () => {
const rows = await fetchSuggestions(q, controller.signal);
if (controller.signal.aborted) return;
setSuggestions(rows);
setActiveIndex(-1);
setOpen(rows.length > 0);
}, DEBOUNCE_MS);
return () => {
clearTimeout(timer);
controller.abort();
};
}, [query, enabled]);
return {
suggestions,
open,
activeIndex,
setActiveIndex,
close: () => {
dismissed.current = query.trim();
setOpen(false);
setActiveIndex(-1);
},
};
}
```
- [ ] **Step 5: Run the tests to verify they pass**
Run: `cd nextjs-app && npx jest __tests__/hooks/useSchoolSuggest.test.tsx && npx tsc --noEmit`
Expected: PASS, 5 tests, typecheck clean.
- [ ] **Step 6: Commit**
```bash
git add nextjs-app/lib/suggest.ts nextjs-app/hooks/useSchoolSuggest.ts nextjs-app/__tests__/hooks/useSchoolSuggest.test.tsx
git commit -m "feat(suggest): debounced, abortable suggestion hook"
```
---
### Task 5: The list, with its ARIA
**Files:**
- Create: `nextjs-app/components/SuggestList.tsx`
- Create: `nextjs-app/components/SuggestList.module.css`
- Create: `nextjs-app/__tests__/components/SuggestList.test.tsx`
**Interfaces:**
- Consumes: `Suggestion` from `@/lib/suggest`
- Produces: `<SuggestList id, suggestions, activeIndex, onPick, onHover />`, and `suggestOptionId(id: string, index: number): string`
- [ ] **Step 1: Write the failing test**
Create `nextjs-app/__tests__/components/SuggestList.test.tsx`:
```tsx
import { render, screen } from '@testing-library/react';
import { SuggestList, suggestOptionId } from '@/components/SuggestList';
const ROWS = [
{ urn: 1, school_name: "St Mary's Primary", local_authority: 'Camden',
postcode: 'NW1 1AA', phase: 'Primary', school_type: 'Voluntary aided school' },
{ urn: 2, school_name: "St Mary's Primary", local_authority: 'Barnet',
postcode: 'EN5 2AA', phase: 'Primary', school_type: 'Community school' },
];
describe('SuggestList', () => {
it('is a listbox of options', () => {
render(<SuggestList id="s" suggestions={ROWS} activeIndex={-1}
onPick={() => {}} onHover={() => {}} />);
expect(screen.getByRole('listbox')).toBeInTheDocument();
expect(screen.getAllByRole('option')).toHaveLength(2);
});
it('shows the local authority, which is what tells two schools apart', () => {
// Both rows are "St Mary's Primary". Without the authority the list is
// unusable for exactly the query autosuggest exists to serve.
render(<SuggestList id="s" suggestions={ROWS} activeIndex={-1}
onPick={() => {}} onHover={() => {}} />);
expect(screen.getByText('Camden')).toBeInTheDocument();
expect(screen.getByText('Barnet')).toBeInTheDocument();
});
it('marks only the active option selected', () => {
render(<SuggestList id="s" suggestions={ROWS} activeIndex={1}
onPick={() => {}} onHover={() => {}} />);
const options = screen.getAllByRole('option');
expect(options[0]).toHaveAttribute('aria-selected', 'false');
expect(options[1]).toHaveAttribute('aria-selected', 'true');
});
it('gives each option the id the input will point at', () => {
// aria-activedescendant on the input has to name a real element id, or
// a screen reader announces nothing as the user arrows through.
render(<SuggestList id="s" suggestions={ROWS} activeIndex={0}
onPick={() => {}} onHover={() => {}} />);
expect(screen.getAllByRole('option')[0]).toHaveAttribute(
'id', suggestOptionId('s', 0));
});
it('renders nothing when there is nothing to suggest', () => {
const { container } = render(<SuggestList id="s" suggestions={[]}
activeIndex={-1} onPick={() => {}} onHover={() => {}} />);
expect(container).toBeEmptyDOMElement();
});
});
```
- [ ] **Step 2: Run it to verify it fails**
Run: `cd nextjs-app && npx jest __tests__/components/SuggestList.test.tsx`
Expected: FAIL — `Cannot find module '@/components/SuggestList'`
- [ ] **Step 3: Write the component**
Create `nextjs-app/components/SuggestList.tsx`:
```tsx
'use client';
/**
* The autosuggest dropdown. Presentational only — it fetches nothing and owns
* no state, so the fetching rules and the ARIA rules can be read separately.
*/
import type { Suggestion } from '@/lib/suggest';
import styles from './SuggestList.module.css';
/** The id the input's aria-activedescendant points at. */
export function suggestOptionId(id: string, index: number): string {
return `${id}-option-${index}`;
}
interface Props {
/** Shared with the input's aria-controls. */
id: string;
suggestions: Suggestion[];
activeIndex: number;
onPick: (s: Suggestion) => void;
onHover: (index: number) => void;
}
export function SuggestList({ id, suggestions, activeIndex, onPick, onHover }: Props) {
if (suggestions.length === 0) return null;
return (
<ul className={styles.list} id={id} role="listbox">
{suggestions.map((s, i) => (
<li
key={s.urn}
id={suggestOptionId(id, i)}
role="option"
aria-selected={i === activeIndex}
className={`${styles.option} ${i === activeIndex ? styles.active : ''}`}
/*
* onMouseDown, not onClick: the input's blur handler closes the list,
* and blur fires before click — so a click handler never runs. This
* is the classic autosuggest bug where the dropdown is unclickable
* with a mouse while working perfectly with a keyboard.
*/
onMouseDown={(e) => { e.preventDefault(); onPick(s); }}
onMouseEnter={() => onHover(i)}
>
<span className={styles.name}>{s.school_name}</span>
{/* Not decoration: there are many "St Mary's". */}
<span className={styles.meta}>{s.local_authority}</span>
</li>
))}
</ul>
);
}
```
- [ ] **Step 4: Write the stylesheet**
Create `nextjs-app/components/SuggestList.module.css`:
```css
/* Anchored to the search field's wrapper, which is position: relative. */
.list {
position: absolute;
top: calc(100% + 4px);
left: 0;
right: 0;
z-index: 40;
margin: 0;
padding: 4px;
list-style: none;
max-height: 320px;
overflow-y: auto;
background: var(--color-surface, #fff);
border: 1px solid var(--color-border, #d8d8d8);
border-radius: 10px;
box-shadow: 0 10px 30px rgb(0 0 0 / 12%);
}
.option {
display: flex;
align-items: baseline;
justify-content: space-between;
gap: 12px;
padding: 10px 12px;
border-radius: 6px;
cursor: pointer;
}
/* Hover and keyboard share one style: the active option is the active option
however it became active. */
.option:hover,
.active {
background: var(--color-surface-hover, #f1f1f1);
}
.name {
font-weight: 500;
}
.meta {
font-size: 0.85em;
color: var(--color-text-muted, #666);
white-space: nowrap;
}
```
- [ ] **Step 5: Run the tests to verify they pass**
Run: `cd nextjs-app && npx jest __tests__/components/SuggestList.test.tsx && npx tsc --noEmit`
Expected: PASS, 5 tests, typecheck clean.
- [ ] **Step 6: Commit**
```bash
git add nextjs-app/components/SuggestList.tsx nextjs-app/components/SuggestList.module.css nextjs-app/__tests__/components/SuggestList.test.tsx
git commit -m "feat(suggest): the dropdown, with combobox ARIA"
```
---
### Task 6: Wire it into the search box, behind the flag
**Files:**
- Modify: `backend/flags.py`
- Modify: `nextjs-app/app/page.tsx`
- Modify: `nextjs-app/components/HomeView.tsx`
- Modify: `nextjs-app/components/FilterBar.tsx`
- Modify: `nextjs-app/components/FilterBar.module.css`
- Create: `nextjs-app/__tests__/components/FilterBarSuggest.test.tsx`
**Interfaces:**
- Consumes: `useSchoolSuggest`, `SuggestList`, `suggestOptionId`, `schoolUrl` from `@/lib/utils`, `getFlags` from `@/lib/flags`.
- [ ] **Step 1: Declare the flag**
In `backend/flags.py`, add to the `REGISTRY` tuple, after the
`admission_distance` entry:
```python
Flag(
name="school_autosuggest",
description=(
"School name suggestions as you type in the main search box."
),
added=date(2026, 8, 26),
),
```
- [ ] **Step 2: Write the failing test**
Create `nextjs-app/__tests__/components/FilterBarSuggest.test.tsx`:
```tsx
import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { FilterBar } from '@/components/FilterBar';
const push = jest.fn();
jest.mock('next/navigation', () => ({
useRouter: () => ({ push, replace: jest.fn(), prefetch: jest.fn() }),
usePathname: () => '/',
useSearchParams: () => new URLSearchParams(),
}));
const FILTERS = {
local_authorities: [], school_types: [], years: [], phases: [],
genders: [], admissions_policies: [],
};
const realFetch = global.fetch;
beforeEach(() => {
global.fetch = jest.fn(async () => ({
ok: true,
json: async () => ({ suggestions: [{
urn: 100010, school_name: 'Brecknock Primary School',
local_authority: 'Camden', postcode: 'NW1 1AA',
phase: 'Primary', school_type: 'Community school' }] }),
})) as unknown as typeof fetch;
push.mockClear();
});
afterEach(() => { global.fetch = realFetch; });
describe('FilterBar autosuggest', () => {
it('is a combobox only when the flag is on', () => {
const { rerender } = render(<FilterBar filters={FILTERS} autosuggest={false} />);
expect(screen.queryByRole('combobox')).not.toBeInTheDocument();
rerender(<FilterBar filters={FILTERS} autosuggest />);
expect(screen.getByRole('combobox')).toBeInTheDocument();
});
it('makes no request while the flag is off', async () => {
// Off means off: no listener, no fetch, no markup.
render(<FilterBar filters={FILTERS} autosuggest={false} />);
await userEvent.type(screen.getByPlaceholderText(/School name or postcode/i),
'brecknock');
expect(global.fetch).not.toHaveBeenCalled();
});
it('shows suggestions and navigates when one is chosen', async () => {
render(<FilterBar filters={FILTERS} autosuggest />);
await userEvent.type(screen.getByRole('combobox'), 'brecknock');
const option = await screen.findByRole('option', { name: /Brecknock/ });
await userEvent.click(option);
expect(push).toHaveBeenCalledWith(
expect.stringContaining('/school/100010'));
});
it('suppresses suggestions once the value is a postcode', async () => {
// The box takes a name OR a postcode; suggestions must get out of the way.
//
// fireEvent.change, not userEvent.type: typing sets "N", "NW", "NW1"... and
// "NW1" is not a postcode, so a request for it is correct behaviour. Only
// the settled value is the assertion, so set it in one go.
render(<FilterBar filters={FILTERS} autosuggest />);
fireEvent.change(screen.getByRole('combobox'), { target: { value: 'NW1 1AA' } });
await new Promise((r) => setTimeout(r, 300)); // past the 200ms debounce
expect(global.fetch).not.toHaveBeenCalled();
});
it('Enter with no active option still submits the free-text search', async () => {
// The existing behaviour is preserved, not replaced.
render(<FilterBar filters={FILTERS} autosuggest />);
const input = screen.getByRole('combobox');
await userEvent.type(input, 'brecknock{Enter}');
// updateURL pushes inside startTransition, so the call is not synchronous.
await waitFor(() => expect(push).toHaveBeenCalledWith(
expect.stringContaining('search=brecknock')));
});
});
```
- [ ] **Step 3: Run it to verify it fails**
Run: `cd nextjs-app && npx jest __tests__/components/FilterBarSuggest.test.tsx`
Expected: FAIL — `autosuggest` is not a prop, so no combobox is rendered.
- [ ] **Step 4: Wire the hook into FilterBar**
In `nextjs-app/components/FilterBar.tsx`:
Add to the imports:
```typescript
import { useSchoolSuggest } from "@/hooks/useSchoolSuggest";
import { SuggestList, suggestOptionId } from "./SuggestList";
import { schoolUrl } from "@/lib/utils";
import type { Suggestion } from "@/lib/suggest";
```
Add to `FilterBarProps`:
```typescript
/** Server-read feature flag. Off means no listener, no fetch, no markup. */
autosuggest?: boolean;
```
Add `autosuggest = false,` to the destructured parameters of `FilterBar`.
Inside the component, after the `omniValue` state declaration:
```typescript
const suggestId = `school-suggest-${isHero ? "hero" : "bar"}`;
// Suppressed once the value parses as a postcode: the box takes a school
// name OR a postcode, and suggesting schools during postcode entry fights
// the user rather than helping them.
const suggestEnabled = autosuggest && !isValidPostcode(omniValue);
const { suggestions, open, activeIndex, setActiveIndex, close } =
useSchoolSuggest(omniValue, suggestEnabled);
const pickSuggestion = (s: Suggestion) => {
close();
track('search_submitted', {
query: s.school_name.toLowerCase(),
via: 'suggestion',
urn: s.urn,
has_postcode: false,
filters_active: '',
filters_count: 0,
});
router.push(schoolUrl(s.urn, s.school_name));
};
const handleOmniKeyDown = (e: React.KeyboardEvent<HTMLInputElement>) => {
if (!open) return;
if (e.key === "ArrowDown") {
e.preventDefault();
setActiveIndex(activeIndex + 1 >= suggestions.length ? 0 : activeIndex + 1);
} else if (e.key === "ArrowUp") {
e.preventDefault();
setActiveIndex(activeIndex <= 0 ? suggestions.length - 1 : activeIndex - 1);
} else if (e.key === "Escape") {
close();
} else if (e.key === "Enter" && activeIndex >= 0) {
// Only when an option is active. With none, the event falls through to
// the form's submit handler and searches the typed text, as it does now.
e.preventDefault();
pickSuggestion(suggestions[activeIndex]);
}
};
```
- [ ] **Step 5: Wire the markup**
In `nextjs-app/components/FilterBar.tsx`, replace the existing omni `<input>`
element with:
```tsx
<input
ref={inputRef}
type="search"
value={omniValue}
onChange={(e) => setOmniValue(e.target.value)}
onKeyDown={handleOmniKeyDown}
onBlur={close}
placeholder="School name or postcode"
className={styles.omniInput}
{...(autosuggest ? {
role: "combobox",
"aria-expanded": open,
"aria-controls": suggestId,
"aria-autocomplete": "list" as const,
"aria-activedescendant":
activeIndex >= 0 ? suggestOptionId(suggestId, activeIndex) : undefined,
autoComplete: "off",
} : {})}
/>
```
And directly after the closing `</button>` of the search submit button, still
inside the wrapper element that holds the input:
```tsx
{autosuggest && open && (
<SuggestList
id={suggestId}
suggestions={suggestions}
activeIndex={activeIndex}
onPick={pickSuggestion}
onHover={setActiveIndex}
/>
)}
```
- [ ] **Step 6: Anchor the dropdown**
The list is `position: absolute`, so its nearest positioned ancestor must be
the wrapper around the input and the submit button. That is
`.omniBoxContainer`, and it **does not currently declare `position: relative`**
— without this the dropdown anchors to the page and lands in the wrong place.
In `nextjs-app/components/FilterBar.module.css`, change:
```css
.omniBoxContainer {
display: flex;
align-items: center;
gap: 0.5rem;
}
```
to:
```css
.omniBoxContainer {
display: flex;
align-items: center;
gap: 0.5rem;
/* The suggestion dropdown is absolutely positioned against this box. */
position: relative;
}
```
- [ ] **Step 7: Thread the flag from the server**
In `nextjs-app/app/page.tsx`, add the import:
```typescript
import { getFlags } from '@/lib/flags';
```
Inside `HomePage`, before the `try`:
```typescript
// Server-read: no flag value reaches the browser bundle.
const flags = await getFlags();
const autosuggest = flags.school_autosuggest === true;
```
Add `autosuggest={autosuggest}` to **both** `<HomeView ... />` render sites —
the success path and the `catch` fallback. Missing the second means the flag
silently does nothing whenever the API call fails.
In `nextjs-app/components/HomeView.tsx`, add `autosuggest?: boolean` to its
props interface, destructure it with a default of `false`, and pass
`autosuggest={autosuggest}` to **both** `<FilterBar ... />` instances — hero
and sticky.
- [ ] **Step 8: Run the tests to verify they pass**
Run: `cd nextjs-app && npx jest && npx tsc --noEmit && npm run build`
Expected: all pass, typecheck clean, build green.
- [ ] **Step 9: Run the backend suite**
The flag registry gained an entry, so the registry tests must still pass.
Run: `uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests -q`
Expected: all pass.
- [ ] **Step 10: Commit**
```bash
git add backend/flags.py nextjs-app/app/page.tsx nextjs-app/components/HomeView.tsx nextjs-app/components/FilterBar.tsx nextjs-app/components/FilterBar.module.css nextjs-app/__tests__/components/FilterBarSuggest.test.tsx
git commit -m "feat(suggest): wire autosuggest into the search box behind a flag"
```
---
### Task 7: E2E journeys
**Files:**
- Modify: `e2e/tests/journeys.spec.ts`
- [ ] **Step 1: Write the journeys**
`/api/flags` is denied to the public, so feature state is read from its
observable effect — the same approach the distance journeys use.
Append to `e2e/tests/journeys.spec.ts`:
```typescript
/*
* School autosuggest (spec 2026-08-26).
*/
async function autosuggestIsOn(page: Page): Promise<boolean> {
await page.goto('/');
return (await page.getByRole('combobox').count()) > 0;
}
test('the suggest endpoint answers from Typesense', async ({ page }) => {
// Not flagged — the endpoint is live even while the UI is dark, so it can
// be smoke-tested before the feature is switched on.
const res = await page.request.get('/api/suggest?q=brecknock');
expect(res.ok()).toBeTruthy();
const { suggestions } = await res.json();
expect(Array.isArray(suggestions)).toBeTruthy();
if (suggestions.length) {
// Local authority is what tells two "St Mary's" apart.
expect(suggestions[0]).toHaveProperty('school_name');
expect(suggestions[0]).toHaveProperty('local_authority');
}
});
test('a one-character query is answered, not rejected', async ({ page }) => {
// The keystroke path never errors on ordinary input.
const res = await page.request.get('/api/suggest?q=b');
expect(res.status()).toBe(200);
expect((await res.json()).suggestions).toEqual([]);
});
test('the suggest response is cacheable', async ({ page }) => {
const res = await page.request.get('/api/suggest?q=brecknock');
expect(res.headers()['cache-control'] ?? '').toContain('s-maxage');
});
test('typing a school name suggests it, and choosing it opens that school', async ({ page }) => {
test.skip(!(await autosuggestIsOn(page)),
'the school_autosuggest flag is off in this environment');
// A school certain to exist in any environment with data.
const { schools } = await (await page.request.get('/api/schools?page_size=1')).json();
test.skip(!schools?.length, 'no schools in this environment');
const name = schools[0].school_name as string;
await page.goto('/');
await page.getByRole('combobox').first().fill(name.slice(0, 12));
const option = page.getByRole('option').first();
await expect(option).toBeVisible();
await option.click();
await expect(page).toHaveURL(/\/school\/\d+/);
});
test('with autosuggest off, the search box is a plain input', async ({ page }) => {
test.skip(await autosuggestIsOn(page),
'the school_autosuggest flag is on in this environment');
await page.goto('/');
await expect(page.getByRole('combobox')).toHaveCount(0);
// And the box still works: the existing search must be untouched.
await page.getByPlaceholder(/School name or postcode/i).first().fill('abbey');
await page.getByRole('button', { name: /Search/i }).first().click();
await expect(page).toHaveURL(/search=abbey/);
});
```
- [ ] **Step 2: Verify the spec compiles and the tests are collected**
Run: `cd e2e && npx playwright test --list`
Expected: the five new titles appear; total rises by 5.
- [ ] **Step 3: Commit**
```bash
git add e2e/tests/journeys.spec.ts
git commit -m "test(e2e): autosuggest journeys, gated on the flag"
```
---
### Task 8: Verify and open the PR
- [ ] **Step 1: Run everything**
```bash
uv run --quiet --with-requirements requirements.txt --with pytest --with "httpx==0.27.0" python -m pytest backend/tests -q
cd nextjs-app && npx jest && npx tsc --noEmit && npm run build
cd ../e2e && npx playwright test --list
```
Expected: backend green, frontend green, typecheck clean, build green, e2e collects.
- [ ] **Step 2: Open the PR against `main`**
The body must state: the rate-limit keying change applies unflagged and
site-wide; autosuggest itself is off until toggled in Unleash; and that
`/api/suggest` is live regardless, deliberately.
## Notes for the executor
**Task 1 changes rate limiting for every endpoint.** It is the one change here
that is not behind a flag, and it is the one worth the most review attention.
Before it, everyone shares one 60/minute bucket; after it, each caller gets
their own, bounded by a global ceiling. The header it keys on is only
trustworthy for traffic that actually passed through Cloudflare, which this
app cannot verify — closing that needs Authenticated Origin Pulls or an origin
firewall, and is the most valuable follow-up in the spec's Risks.
**The in-app global ceiling is required, and slowapi cannot express it.**
Code review found that `client_key` trusts `CF-Connecting-IP` with no way to
verify the request reached the origin through Cloudflare — so rotating that
header mints a fresh bucket per request and defeats per-client limits entirely,
which is worse against abuse than the shared bucket it replaced. The ceiling
(`GlobalRateLimitMiddleware`) bounds that, and is written by hand because
slowapi's `default_limits` and `application_limits` are both keyed by
`key_func` — per-client, not global — and the latter needs `SlowAPIMiddleware`,
which this app does not install. See spec §1.1.
**`onMouseDown`, not `onClick`, on the options.** The input's `onBlur` closes
the list and blur fires first, so a click handler never runs. This is the
classic bug where the dropdown works by keyboard and is dead to the mouse.
**Both render sites, twice over.** `page.tsx` renders `HomeView` in the success
path *and* the catch fallback; `HomeView` renders `FilterBar` as hero *and*
sticky. Missing any of the four makes the flag silently do nothing somewhere.