Server-side bot tracking

Most crawlers never execute JavaScript: Googlebot sometimes renders pages, but GPTBot, ClaudeBot, PerplexityBot, CCBot and the AI assistants reading a page on demand (ChatGPT-User, Claude-User, Perplexity-User) usually don't. To measure them, send Eureka a hit from your server or your CDN for every HTML page served. This fills the Indexing bots — SEO & AI, Pages read by AI assistants and Verified vs spoofed bots reports.

Available on the Pro and Scale plans.

1. Create an ingest key

Admin → Properties → Ingest keys → New key. The key is shown once. Keep it secret: it goes in your server or CDN configuration, never in a page.

2a. Cloudflare Worker (no latency added)

The worker forwards one hit per HTML response, in the background (waitUntil).

// Env vars: AH_COLLECTOR (https://collect.example.com), AH_PID (AH-XXXXXX), AH_INGEST_KEY (secret)
const FORWARDED_HEADERS = ["accept", "accept-language", "from", "signature-agent", "signature-input", "signature", "sec-ch-ua", "sec-fetch-mode"];

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const type = response.headers.get("content-type") || "";
    if (request.method === "GET" && type.includes("text/html")) {
      ctx.waitUntil(sendHit(request, response, env).catch(() => {}));
    }
    return response;
  },
};

async function sendHit(request, response, env) {
  const h = {};
  for (const name of FORWARDED_HEADERS) {
    const v = request.headers.get(name);
    if (v) h[name] = v.slice(0, 512);
  }
  await fetch(`${env.AH_COLLECTOR}/v1/s`, {
    method: "POST",
    headers: { "content-type": "application/json", authorization: `Bearer ${env.AH_INGEST_KEY}` },
    body: JSON.stringify({
      v: 1, pid: env.AH_PID,
      hits: [{
        ts: Date.now(), ip: request.headers.get("cf-connecting-ip") || "",
        ua: request.headers.get("user-agent") || "", m: request.method, url: request.url,
        st: response.status, ref: request.headers.get("referer") || "", h,
      }],
    }),
  });
}

The full version (with Cloudflare's verified-bot flag and ASN) is in the repository at deploy/edge/cloudflare-worker.js. The same approach works with Fastly Compute and Vercel middleware.

2b. nginx + Vector

Log HTML responses as JSON, and ship them to the collector with Vector:

log_format ah escape=json '{"ts":"$msec","ip":"$remote_addr","ua":"$http_user_agent","m":"$request_method",'
  '"url":"$scheme://$host$request_uri","st":$status,"ref":"$http_referer","bytes":$body_bytes_sent,'
  '"sa":"$http_signature_agent","ct":"$sent_http_content_type"}';
access_log /var/log/nginx/ah.json.log ah;

The Vector configuration (deploy/edge/nginx-vector.toml) keeps HTML responses only and posts batches of up to 500 hits to https://collect.example.com/v1/s?pid=AH-XXXXXX with Authorization: Bearer ${AH_INGEST_KEY}. Fluent Bit or any log shipper that can POST JSON works the same way.

The endpoint

POST {collector}/v1/s, Authorization: Bearer <ingest key>. Body: {"v":1,"pid":"AH-XXXXXX","hits":[…]}, or a bare JSON array of hits with ?pid= in the URL.

Hit field	Meaning
ts	Time, epoch milliseconds
ip	Client IP (used for verification and geography, then truncated; never stored)
ua	User agent
m, url, st, ref, bytes	Method, full URL, HTTP status, referrer, size
h	Selected request headers: signature-agent, signature-input, signature, from, accept…

Response 202 {"accepted": n}. Under load the collector answers 503 with the number accepted: retry the batch, duplicates are absorbed.

Hits from humans are dropped (the tag already measures them), and a server hit matching a page view the tag sent in the last 30 seconds is not counted twice.