How AI traffic is measured

Everything the AI traffic tab shows comes from your own server: no polling, no simulated questions put to ChatGPT, no guessed score. This page describes exactly what is observed, what is verified, and above all what the measurement does not prove.

#Three stages, three different signals

The three stages never add up: they are three distinct populations. Bots are never counted among your visitors.

#What the measurement does not prove

This is the part most tools keep quiet about, so let us put it in writing.

A live fetch proves that a system retrieved your page while answering someone. It does not prove that:

• the final answer cited your page;
• your content influenced that answer;
• the person saw your name;
• the question asked had any commercial bearing on your business.

Likewise, we never know the question that triggered a visit or a fetch: it is not transmitted anywhere. That is why the interface says "landing pages" and "fetched live", never "cited".

Probable arrivals — those inferred from a text fragment in the URL, the typical signature of a click from an AI answer with no referrer — are counted separately and never merged with what is measured.

#How bot identity is verified

A user-agent is a claim: any program can call itself GPTBot, and plenty of scrapers do exactly that to slip past blocks. Trusting the user-agent alone means measuring what you are told.

So Snorklee checks the IP address of every hit against the ranges published by the vendors themselves:

Verification happens bot by bot, not vendor by vendor: GPTBot and ChatGPT-User publish different ranges, so an address that is valid for one does not validate the other. The lists are re-downloaded every night; if a download fails, the previous version stays in place.

Three possible verdicts, shown under the crawler table:

• Verified — the address belongs to the ranges published by that bot's vendor.
• Declared — that vendor publishes no ranges (Common Crawl, Bytespider, Amazonbot…). All we can do is record what the bot claims, and we never present it as verified.
• Discarded — the bot claimed a vendor from an address that is not theirs. Those hits are removed from every counter in the tab; only the spoofing total remains visible, so you know what was set aside.

#Where the signal is captured

Bots only read raw HTML: they do not run JavaScript and are therefore invisible to a classic measurement script. The w.js tag alone will never see them.

Capturing the "Crawled" stage requires a server-side signal — the crawler beacon (WordPress plugin, Next.js middleware, Cloudflare worker, PHP snippet…). Without it, only human visits from AI chats are measured. The beacon passes the bot's address along so the verification described above can happen.

#What the bots read, page by page

Every hit carries the exact token the bot claims (GPTBot, ChatGPT-User…), not just its vendor name. That is what separates two opposite facts about the same page: harvesting for model training, and a live fetch while an AI answers someone.

The “Who reads what” card therefore crosses bot × page, both ways. Counts there are exact, with no threshold and no rounding: those rows contain no human visitor data, so no anonymisation floor applies — a bot is not a person. Counters that mix humans and bots remain subject to the rules described in Data collected.

Tokens have only been recorded since August 2026: earlier hits stay attributed to their vendor, without per-bot detail.

#robots.txt compliance

Every night Snorklee reads your site's robots.txt and compares it with what bots actually fetched. Since robots.txt rules are written per token, the comparison happens at that same level.

The result is only shown on a complete fact:

• File not read (timeout, server error, unreachable domain) → nothing is asserted. A missing answer is never presented as compliance.
• No robots.txt → we say exactly that: no rule can be held against the bots.
• Unidentifiable bot → its hits are excluded from the calculation and reported separately, never counted as compliant.

When a violation is found, the page involved and the exact line of the file forbidding it are quoted, so the finding can be verified.

#Generate your robots.txt

From the AI Traffic tab, Snorklee produces a robots.txt built from the bots that actually visited your site — not a generic list. You tick the ones you want to keep out, copy the text, paste it at your host.

By default, only training collection is blocked. Bots that index (search) or fetch a page live to answer someone (assistant) send you visitors: blocking them would cut that traffic. Snorklee does not forbid it, but it warns you.

Only bots actually observed are pre-ticked: a rule aimed at a bot that never came makes the file longer without protecting anything.

Two outputs: the block to add (your existing rules are never touched), or the full file, merged with the one already in place — your Disallow and Sitemap lines are kept. Running the generator again replaces the previous Snorklee block instead of stacking another one.

What the generator does not do. Snorklee does not serve your robots.txt and never edits it: until you paste the text, nothing has changed. And a robots.txt remains a polite request, not a barrier — which is exactly why the compliance check described above exists: it tells you who ignores it.

#Privacy

The IP address is used only, at request time, to geolocate at country level, classify traffic and verify bot identity. It is never stored in events — see Data collected.