The “Competitor comparison” card on a scan report shows a handful of third-party pages that, in our judgement, rank for the same topic as the page you scanned. We then diff their structured data against yours and flag the schema types they emit but you don’t.

To pick those pages we issue a single search query to Serper.dev, which is a thin wrapper around the public Google Search results page. The query we send is built from your page signals in the following strict priority order:

1. NLP topics— if the local Ollama model successfully tagged the page with two or more topical keywords, we use up to the top five.
2. Page title — the contents of <title>, with common site-name suffixes (“ | Brand”, “ – Brand”) stripped. If the resulting title is at least 12 characters we use it verbatim and append up to two NLP topics for focus.
3. First H1— if there is no usable title, we fall back to the page’s first <h1> text.
4. NLP summary keywords— as a last resort, we extract three to six topical keywords from the LLM-generated page summary (filtering out structural HTML tokens and English stopwords).

If none of those four steps produces a usable query, we deliberately skip the SERP call entirely. The “Competitor comparison” card will say so, and a warning will appear on the scan explaining which signal was missing. We prefer an honest empty state over a fabricated answer.

To make this fully verifiable, the exact query we sent is printed directly on the scan reportunder the competitor table. If the query reads sensibly for your page, you should expect the results to be useful; if it does not, the underlying problem is almost always NLP enrichment failing — run the scan again once your local LLM is warm, and the results will improve.

Search engine results are not deterministic. Two scans of the same page minutes apart can return slightly different competitor URLs for several legitimate reasons:

• Google personalises results.The Serper API deliberately fetches a depersonalised view from a clean datacentre, but Google’s index changes hour to hour as pages are crawled, ranked, and re-ranked.
• The query itself can change. If a previous scan used NLP topics but a later one fell back to the page title (because Ollama was cold), the queries are different and Google will rank different pages.
• Regional weighting.Serper’s default is US-English; results for your audience’s region may differ.
• Index churn. Pages get added, removed, and redirected continuously. A competitor that ranked in the top ten yesterday may not today.

This is a property of public search engines — not a bug in Schema Intelligence. The signal we offer is directional(“here are pages ranking for your topic that emit FAQPage; you don’t”), not a guarantee that the exact same three URLs will be returned tomorrow.

Every scan goes through the same deterministic pipeline. Failures in any one step degrade the output of that step but never crash the scan.

1. Crawl. We fetch the URL with a clearly identified user-agent, honour robots.txt, follow up to three redirects, reject private IP ranges, and read at most a few megabytes of body.
2. Schema parse. We extract every JSON-LD, Microdata, and RDFa entity in the page, deduplicating identical copies and flagging conflicts.
3. DOM analysis.We look for the heuristic fingerprints of structured-data opportunities — FAQ question patterns, How-To step sequences, video embeds, product markers, rating widgets, table density, heading hierarchy, recipe ingredient/step layouts, event date-and-venue blocks, local-business contact details, job-posting fields, and other vertical fingerprints.
4. NLP enrichment(optional). When a local Ollama instance is reachable, we send the page’s body text to it and ask for structured topics, entities, intent, semantic sections, and a one-paragraph summary. Output is validated against a strict JSON schema before we trust any of it.
5. Recommendations. A rules engine combines the previous steps to surface missing schema types, with an associated confidence and severity.
6. Validation.Each detected entity is checked against a curated subset of schema.org constraints (required fields, invalid nesting, deprecated properties) plus vertical quality checks (e.g. an Offer with no price, a Product with no image, a LocalBusiness missing geo/hours). Types we don’t model in depth are accepted and evaluated generically rather than flagged as errors.
7. Scoring. We assign two numeric scores: coverage(how much of the structured data the page’s content actually calls for is present, measured against a content-derived target model) and AI readiness(a weighted average of criteria that actually apply to the detected page type — for instance, the “recipe has ingredients” criterion is omitted entirely for an article).
8. Competitor analysis (optional). See section 1.

• Schema.org vocabulary — sourced from schema.org, licensed CC BY-SA 3.0.
• Wikidata — entity QIDs, labels, descriptions, and sameAs URIs are queried from wikidata.org, released under CC0.
• Wikipedia— companion Wikipedia URLs are derived from Wikidata sitelinks. Wikipedia content is licensed CC BY-SA 4.0.
• Serper.dev— commercial third-party SERP API. Usage is governed by Serper’s terms. We never send page bodies, user information, or personally-identifiable data to Serper — only the generated query string described in section 1.
• Ollama— the LLM runs entirely on infrastructure you control (typically your laptop or your server). No page content leaves your machine for AI processing.

What we store.For every scan we keep: the URL you submitted, the page’s public HTTP response, the parsed structured data, the derived signals, the recommendations, and the generated NLP output. We retain these for diagnostics and history comparisons.

What we do not store. We do not collect personal information about visitors directly, do not set our own first-party tracking cookies, do not log request bodies sent to your scanned pages, and do not store API credentials in plaintext (API keys are Argon2id-hashed before they touch the database). Google Analytics 4 (see sub-processors below) sets its standard measurement cookies on the public site; opt out via your browser settings or an extension such as the official Google Analytics Opt-out browser add-on.

Sub-processors used in production. Schema Intelligence relies on a small number of vendors to operate the public site. None of them receive your scan content unless explicitly noted in section 4.

• Vercel— serves the web frontend and runs Vercel Analytics + Speed Insights for aggregate visit counts and Web Vitals. Both are cookieless and use a daily-rotated, salted hash of the request IP + user-agent as a visitor identifier; no personal data is stored.
• Google Analytics 4— page-view and traffic measurement on the public site. When the NEXT_PUBLIC_GA4_MEASUREMENT_ID environment variable is set, the GA4 loader script runs on every public page and stores the standard first-party _ga*cookies in your browser. Schema Intelligence does not configure GA4 Consent Mode during the beta — a cookie-consent banner that lets you toggle analytics is planned for a later release. In the meantime you can opt out using your browser’s built-in tracking-protection settings, an ad/tracker-blocking extension, or the official Google Analytics Opt-out browser add-on.
• Fly.io— hosts the API and worker processes, plus the Postgres and Redis they depend on. Scan rows live in this Postgres instance.
• Sentry— captures unhandled exceptions and performance traces. PII scrubbing is enabled; we do not send page bodies or scan results to Sentry.
• Groq Cloud— OpenAI-compatible LLM provider used for NLP enrichment and JSON-LD snippet generation. Each call sends an excerpt of the scanned page’s public text and a small structured prompt. Groq does not retain prompt content for training under their commercial terms.

Outbound traffic during a scan.A scan can produce HTTP requests to: the URL you submitted (always); Wikidata’s public REST API (entity enrichment); Groq Cloud (NLP + snippet generation); and Serper.dev (competitor analysis — disabled by default during beta). Each is configurable via .env flags on a self-hosted deployment.

Beta retention policy. Scan inputs (URL, derived results, and an anonymised IP bucket used only for rate limiting) are retained during the beta to inform product improvements. Beta data is not shared or sold. After the beta we will introduce a rolling retention window (targeted: 90 days) once user accounts ship and per-user scope is available.

Schema Intelligence is a recommendation tool. Results are generated using NLP and AI, which can make mistakes, so the output of every step is heuristic and should be reviewed and verified before you publish schema or ship changes to production:

• AI-generated JSON-LD snippetsare produced by a local LLM grounded in your page’s text. We validate the JSON shape and strip top-level keys that don’t belong to schema.org, but you must read the snippet, replace placeholder values, and verify it against your actual page before deployment.
• The AI readiness score reflects our reading of current best practices for AI-search visibility. It is not endorsed, certified, or ranked by Google, Bing, OpenAI, Anthropic, Perplexity, or any other search/AI vendor. A high score is not a guarantee of richer SERP placement.
• Recommendations are based on rules that match observable page patterns to schema.org types. Confidence is expressed numerically; treat anything below 0.6 as an investigation prompt rather than a verdict.
• Validationcovers a curated subset of schema.org rules — not the entire specification. Always cross-check with validator.schema.org and the relevant search engine’s rich-results tool before going live.

By submitting a URL to Schema Intelligence you confirm that:

• You have the right to fetch and analyse the page (it is public and not behind a paywall or authentication wall you don’t own).
• The target server’s robots.txt permits an automated user-agent fetch, or that you accept responsibility for the request regardless. Schema Intelligence reads and respects robots.txt by default.
• You will not use Schema Intelligence to enumerate private or internal IP ranges. The crawler rejects RFC 1918 / loopback / link-local targets unless explicitly enabled in a test environment.
• You will not abuse the rate limits or attempt to evade them by rotating API keys.

Schema.org, Google, Bing, OpenAI, Anthropic, Perplexity, YouTube, Wikipedia, Wikidata, Ollama, Serper, and other product names referenced on this site are trademarks or registered trademarks of their respective owners. Their use here is purely descriptive; Schema Intelligence is not affiliated with, endorsed by, or sponsored by any of them.

Schema Intelligence is provided “as is” without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and non-infringement. In no event shall the authors or operators be liable for any claim, damages, or other liability arising from the use of this software, including but not limited to changes in SEO performance, search engine rankings, revenue, or business decisions made on the basis of its output.

We update this page when methodology, data sources, or licences change. The “Last updated” date at the top of the page reflects the most recent edit. Substantive changes (a new data source, a different SERP provider, a change to how scores are computed) will also be noted in the project README’s release notes. Questions or corrections? Get in touch from the homepage.