Is Your Website in Common Crawl? Why It Matters for AI Visibility

Large language models do not learn about your brand from your website directly. They learn from the corpora they were trained on — and for the open web, the single largest and most widely used corpus is Common Crawl. If your pages are not in it, a whole class of models most likely never saw your content during training. That makes "is my site in Common Crawl?" one of the few training-data questions you can actually answer with public data.

What Common Crawl is

Common Crawl is a nonprofit that has been crawling the web since 2008 and publishing the result as free, open datasets — petabytes of raw page captures. A new crawl ships roughly once a month, each identified by a dated index such as CC-MAIN-2026-30, and each crawl's contents can be queried URL by URL through the public CDX index API.

The crawler behind it is CCBot. It respects robots.txt, discovers pages through links and sitemaps, and captures what the server returns — which means heavily client-rendered content that only appears after JavaScript execution may be captured as an empty shell.

Why being in it matters for AI visibility

Common Crawl is a primary ingredient in LLM training data. OpenAI's GPT-3 paper reports that filtered Common Crawl made up about 60% of the training mix (Brown et al., 2020), and widely used derivative datasets such as C4 and RefinedWeb — behind models like T5, LLaMA and Falcon — are built directly from Common Crawl snapshots.

The honest framing has two sides:

• Presence is a chance, not a guarantee. Vendors filter, deduplicate and sample aggressively; not every model ingests every crawl. A captured page may have entered training data.
• Absence is close to a verdict. If your pages are missing from crawl after crawl, the models trained on this corpus had essentially no opportunity to learn your content from your own site. What they know about you then comes only from third-party pages that mention you.

This is why Common Crawl coverage sits at the foundation of Generative Engine Optimization (GEO): before you optimize how AI systems describe you, confirm that your content exists in what they read.

Common Crawl coverage is not SEO

Google indexing and Common Crawl coverage are separate systems with separate crawlers, discovery paths and filters. A page can rank on the first page of Google and still be absent from Common Crawl — for example because a firewall rule blocks CCBot while allowing Googlebot, or because the site is new and CCBot has not discovered it yet. The reverse also happens. If you have already done a technical AI bot access audit, checking Common Crawl is the natural next line in the same checklist: access is about whether bots can read you; Common Crawl is about whether the open corpus did.

How to check — for free

You can query the CDX API by hand (index.commoncrawl.org), but it is rate-limited, returns raw JSON lines and covers one crawl per query. Maya's free Common Crawl Checker does the aggregation for you — no signup required. Enter a domain and it scans the latest five monthly crawls and reports:

What you see	Why it matters
Unique captured URLs and per-crawl trend	Whether coverage is growing, stable or decaying month over month
Latest capture date	How fresh the corpus's copy of your site is
Section breakdown (top paths)	Which parts of the site the crawler actually reaches — blog but not product pages is a common surprise
Content types and subdomains	Whether captures are real HTML pages or stray assets, and which hosts are covered
HTTP status distribution	Captures of 301s and 404s are wasted coverage
Duplicate-content detection	URLs whose captured content is byte-identical to another URL's
CSV export	The full deduped URL list for your own analysis

Two readings to act on:

• Zero or near-zero URLs: check robots.txt for a CCBot block, verify pages return 200 to non-browser clients, and make sure the main content is present in the server-rendered HTML. Then wait — new coverage appears on the monthly crawl cadence, not overnight.
• Coverage, but the wrong coverage: if captures concentrate on old blog paths while your product and pricing pages are absent, the pages you most want models to learn from are the ones missing. Internal linking and sitemap hygiene usually move this.

Keep the claim honest

Being in Common Crawl does not mean an AI assistant will cite you or recommend your brand — training-data presence, retrieval and citation are different stages. Use the checker to close the first gap, then measure the outcome separately: whether AI answers actually mention and cite you is what Maya tracks continuously across ChatGPT, Claude, Gemini and the rest.