Building an AI SEO engine: Architecture decisions that matter

Most AI content tools demo well because one prompt over one page is easy; the hard part is keeping thousands of URLs, their search data and their drafts in agreement over months.

An AI SEO engine is not a writing tool with a queue in front of it. It is a data system that ends in generation: it must know which pages exist and what state each draft is in.

When one of these systems stalls, the model is rarely the cause. It is a crawl that cannot finish, a join that silently loses rows, or a rerun that overwrites work an editor approved.

Crawling and the URL inventory

The inventory is the spine. Everything downstream keys off a canonical URL record, so the crawler resolves redirects, collapses parameter variants, respects canonical tags and records when each page was last seen.

Each crawl is stored as an append-only observation rather than overwriting the page row, so the system can say when a title changed. Five properties earn their place:

1. A stable identifier that survives slug changes, so a rename keeps its history.
2. The canonical target, so variants roll up into one reporting row.
3. Indexability state, separating noindex, blocked by robots, and simply not linked.
4. A content hash, so a rerun can tell an edit from a re-render.
5. The last successful crawl time, kept distinct from the last attempt.

Without the last two, every downstream job re-fetches a page to decide whether it has work, spending crawl budget on a question the database should already hold.

Joining search data per URL

Search Console, analytics and rank tracking describe the same page with different keys, time grains and rounding. The join must be explicit about all three, or the scores built on it are quietly wrong.

URLs are normalised at write time for scheme, host case, trailing slash and tracking parameters. Query rows aggregate to a per-URL table on a fixed daily grain, and late data is reprocessed on a rolling window.

Rows that fail to match a known URL go to a rejects table rather than being dropped. A rising reject count is the earliest sign that a site change has broken the inventory.

Scoring and prioritisation

Deciding what to write is a ranking problem, and it must be legible to the people whose week it schedules. A score that cannot be explained gets overridden, and an overridden engine is decoration.

Scores are computed from stored inputs on a schedule rather than at read time, so the score a page held on a given day can be reproduced. Each carries its components: impressions without clicks, position band and overlap with a sibling URL.

Thresholds belong in configuration, because the right cut changes with the size of the site and the capacity of the editors. The engine proposes a queue; a person decides how far it runs.

Drafting with an editor in the loop

Generation is the least interesting part of the system and the part most able to cause damage, so it is built as a proposal rather than a publish.

• Drafts as versions: every run writes a new revision, and nothing overwrites a human edit.
• Explicit review states: proposed, in review, approved and published are separate states, each transition recorded.
• Grounded inputs: the prompt carries the page itself and the queries it already ranks for.
• Deterministic fields: titles, meta descriptions and structured data come from code, not the model.
• Publish through the CMS: drafts go through the API a person uses, so permissions and audit still apply.

Queues, retries and idempotency

Every stage runs as a queued job keyed by URL and run date, so a replay produces the same row rather than a duplicate. Idempotency makes it safe to rerun a failed day.

Stages stay independent, so a rate-limited API delays one step rather than the pipeline, and spend is bounded by the queue rather than by a loop. Such a system can be stopped mid-run and restarted.

• ai
• engineering