Cloudflare CEO says the web's future is "pay to crawl" as bots overtake human traffic

Bot traffic has overtaken human traffic on the internet, according to Cloudflare CEO Matthew Prince. Prince says he didn't expect this tipping point until late 2027, but rapid growth in automated traffic from AI agents moved the timeline up fast. The shift happened over the past few months. The data is still rough, but the trend is unmistakable, Prince adds.

Prince notes that bot, crawler, and agent all mean the same thing; the label just depends on whether you see them as good or bad. As for the future of the web, "clearly it's going to be pay to crawl," he writes.

Cloudflare launched a platform last summer that lets site owners gate AI crawlers and charge for access, but it hasn't gained traction. Prince says his company is still working on "protocols and infrastructure to support the volume it'll require." Meanwhile, Google's AI Overviews and AI Mode have racked up billions of users.