Most of My Traffic Was Bots - and My Analytics Counted Every One

The daily digest for this site landed one morning claiming 41 page views. That's a decent day for a personal site, so I went looking for which page was pulling the weight. The answer was that almost none of it was people - and my own analytics had counted every crawler as a visitor.

The assumption baked into JavaScript analytics

The page-view tracker here is the usual thing: a few lines of client-side JavaScript that POST the current path to my API once the page renders. The implicit reasoning is that a script tag only runs in a real browser, so anything that reports a view must be a human.

That reasoning is about a decade out of date. Googlebot has rendered JavaScript since 2015, and Applebot, Bingbot and Baidu's renderer all do the same now. They load the page, run the bundle, fire the beacon, and land in the database looking exactly like a visitor from Vancouver - except from Mountain View.

The logs had what the database didn't

My tracking endpoint recorded path, day, coarse geolocation and a rotating visitor hash. It never stored the user-agent, which meant the data couldn't answer the question I was now asking. Nothing in the table distinguished a person from a crawler, and no query would recover it.

The nginx access log did have it. Every request to the beacon endpoint sat there with its user-agent attached, so the audit became a join between two systems: count the beacon hits in the logs, group by user-agent, and match the daily totals against the database to confirm I was looking at the same traffic.

What was actually visiting

Twelve days, 238 recorded views. Roughly nine in ten were crawlers announcing themselves in the user-agent: Googlebot around 98, Applebot 52, Baiduspider 31, Bingbot 17, plus a scattering of SEO and AI crawlers. The two-day "spike" that caught my eye was Applebot working through the site after I published a new post.

What remained - actual, plausible humans - was about a dozen people across twelve days. Which is roughly what I'd have guessed for a personal site before analytics told me otherwise.

The one I can't catch

One cluster didn't announce anything. Ten different IPs, all on Chinese carrier networks, every one sending the identical current-Chrome user-agent, hitting between 2am and 6am UTC, each landing directly on a different deep page - /faq, /status, /uses, one project page each - with no referrer and no overlap.

That's a distributed crawler running a real headless browser and wearing an ordinary user-agent. No string matching will ever catch it. You'd need per-IP rate analysis or network-range checks, which for ten views a month is not worth building. I count them as people and accept that the number is slightly wrong in a known direction.

Three states, not two

The fix was to classify the user-agent at write time and store the verdict. The interesting decision was what to do about the rows already in the table.

A boolean column defaulting to false would have been easy, and would have quietly asserted that 1,000 historical views were human when I had good evidence most weren't. So the column is nullable, and null means recorded before classification existed - not "person". The digest reports those as unclassified rather than folding them into either number.

It's a small thing that comes up constantly: when you add a dimension to existing data, the old rows don't have an answer, and inventing a default is how a dataset starts lying to you. A third state is honest and costs one IS NULL in a few queries.

The lesson that generalizes

Store the evidence at ingest, not the conclusion. Had I saved the user-agent from day one - a single column I'd have called noise - the entire audit would have been one GROUP BY instead of a cross-system reconstruction from logs that rotate away after two weeks.

Cheap raw signal beats a clean summary you can't interrogate later. You rarely know at write time which question you'll want to ask.

Got analytics telling you something you don't quite believe? Let's talk.