Capturing AI crawlers

AI bots (GPTBot, ClaudeBot, PerplexityBot…) fetch your HTML without running JavaScript. The w.js tag therefore cannot see them: counting the "Crawled" stage of the AI journey requires a small server-side module. This page explains how to install it.

#Why a server-side module

A classic measurement tag runs in the visitor's browser. A bot requests the page and leaves: no JavaScript is executed, no event is sent. The server module fills that gap. It looks at the user-agent of each request and, only when it recognises an AI bot, reports the page to Snorklee.

It never touches human traffic: your visitors are still measured by w.js alone, with no double counting.

#Install the module for your platform

Each file is ready to use, with no dependencies:

• WordPress — nothing to do, the module ships with the official plugin and is on by default.
• Cloudflare — cloudflare-worker.js, deployed as a Worker. Works in front of any site proxied by Cloudflare, including Shopify, Webflow or Wix.
• Next.js / Vercel — vercel-middleware.js, dropped in as middleware.js at the root of the project.
• Netlify — netlify-edge.js, dropped into netlify/edge-functions/ with path /*.
• Node / Express — node-express.js, middleware mounted before your routes.
• PHP — php-generic.php, included at the top of the page outside WordPress.

The module is non-blocking: it never waits for Snorklee's response, and an outage on our side neither slows down nor breaks your page.

#Set your domain

At the top of each file, a single value needs filling in: your domain. It must be identical to the data-site attribute of your w.js snippet.

A different value points to a site Snorklee does not know: reports are then refused and nothing shows up in your dashboard, with no visible error on your site. That is the first thing to check if the counter stays at zero.

#Check that bots are coming through

Open the AI traffic tab. Bot passes appear in the AI crawlers card and in the Who reads what console.

Don't worry if nothing shows up within the hour: bots come at their own pace, sometimes once a week on a small site. Give it several days before concluding something is wrong.

#"The bot's address never reached us"

A user-agent is forged in one line of code. When a bot's vendor publishes its IP ranges, Snorklee checks every hit against them — that is what tells a real GPTBot from a scraper impersonating it.

This check needs the bot's address, which the module passes in a dedicated header. If your dashboard reports hits that "could not be checked", that address is not arriving: the module is in place, but an older version, or an intermediate proxy, lost the information. Take the current version of the file above — this is not spoofing, just an integration to refresh.

#Where it isn't possible

On Wix, Squarespace, Webflow or Framer with native hosting, no code can run server-side on the page request: crawler capture is impossible there. The dashboard says so plainly rather than showing a misleading zero.

The only workaround is to put the domain behind Cloudflare and use the Worker. Human visits coming from AI chats are still measured normally on all of these platforms.