Headless browsers — the behind-the-scenes software program that lets machines surf the net like folks — have been as soon as the area of quality-assurance testers and search engine optimisation companies. However new AI-powered browsers launched this final yr — like Perplexity’s Comet and Browser Firm of New York’s Dia — are bringing new that means to the time period.
These gamers are utilizing headless browsers to energy AI brokers that have to click on, scroll and work together with web sites as a human would, to retrieve data.
For media firms, that raises questions: How a lot visitors is actual vs. automated? How ought to analytics platforms account for agent-driven searching? And for advertisers, it raises considerations about whether or not AI-driven classes will distort measurement or expose new vulnerabilities to fraud.
Learn on to search out out what this implies for publishers, at a time when many are working to dam AI bot visitors and defend their content material from getting scraped with out their management or compensation.
What’s headless searching within the AI period?
Right here’s how folks sometimes work together with AI engines: A consumer goes to an AI software (like ChatGPT for instance) and kinds in a immediate. That AI system both operates on its skilled data, or goes out to retrieve contemporary data from web sites (generally known as retrieval-augmented era.) These AI bots establish themselves to web sites by means of their user-agent strings, comparable to ChatGPT-Person or PerplexityBot. The AI then synthesizes the retrieved data with its personal coaching to generate the ultimate reply to the consumer’s immediate.
However a more moderen wave of AI interfaces — together with autonomous brokers and AI browsers — works in a different way. As a substitute of creating focused retrieval calls and flagging themselves as bots, they depend on headless browsers that truly peruse websites the way in which a human would: clicking, scrolling and loading pages. Typically, these classes current themselves as normal guests (for instance, utilizing a Chromium string) somewhat than as a named bot. That makes it a lot more durable for publishers to differentiate automated searching from actual human visitors, in keeping with a brand new report from Tollbit, a knowledge market for publishers and AI firms, revealed on Sept. 8.
The dynamic may be seen in AI internet browsers. These instruments use AI brokers that function inside a browser to go to web sites, with out the consumer ever needing to see webpages. And somewhat than figuring out themselves by their consumer agent identify (ChatGPT-Person, for instance), they current as a regular customer (comparable to Chromium, Google’s open-source internet browser engine). The consequence: publishers can’t simply inform what’s human and what’s AI-driven.
In response to Tollbit’s report, AI headless browsers can bypass widespread bot-blocking methods. Headless browsers like Browserless declare they will evade these detection instruments, and resolve CAPTCHAs (which are supposed to weed out people from bots).
“Within the case of Perplexity Comet, the agent seems to make use of the consumer’s personal desktop to request webpages when automating workflows. This makes the requests seem to return from a consumer’s residential IP tackle, which may confuse [content delivery network] firms into pondering it’s professional,” Tollbit’s report reads. The report discovered a 336% improve in web sites blocking or redirecting AI bots over the previous yr.
How huge of an issue are AI headless browsers?
It’s nonetheless early days, however these new AI headless browsers do pose a menace to publishers’ companies, execs informed Digiday.
The adoption of AI browsers is simply going to extend. PayPal and Venmo customers (so, hundreds of thousands of individuals) have been simply supplied entry to the Comet browser totally free this month (beforehand, it was out there to Perplexity subscribers and a small group of early customers on a waitlist).
Along with new AI browsers available on the market, OpenAI is rumored to be creating one, and Google is presently creating one by means of its Venture Mariner.
Publishers have already moved to take a stronger stance towards AI bot visitors and content material scraping. AI headless browsers might be the following evolution of that battle.
“There’s the notion that no matter work we’re doing right here on blocking visitors and crawlers and robots… that will get circumvented all collectively,” stated an exec at a enterprise writer, who requested to stay nameless. However the truth that new challenges are cropping up on this space reveals that these efforts are worthwhile, the exec added. “It’s a reinforcement of, we’re doing the precise factor [by] blocking as a lot as we are able to.”
A special exec at a big digital writer, who requested anonymity, stated they believed that, for now, headless browsers posed a minor situation. But it surely might grow to be a major drawback if main gamers like Google or OpenAI undertake them.
What does it imply for publishers?
It’s the following step in a broader shift: AI techniques are beginning to substitute the necessity for customers to go looking or go to writer websites immediately, chipping away at publishers’ (human) visitors. Tollbit knowledge means that human visitors to writer web sites is starting to say no, whereas bot visitors is rising. There was a 9.4% decline within the variety of human guests to web sites between Q1 and Q2 2025, in keeping with the Tollbit report. In the meantime, the ratio of web site visits coming from AI bots in comparison with human guests was 1 in 50, up from 1 in 200 in Q1 2025.
Initially, this can be a transparency situation. Publishers can’t have management over one thing they will’t absolutely see or measure. It’s additionally a enterprise situation. Adverts are being served on publishers’ websites to AI brokers, somewhat than people. In response to Toshit Panigrahi, co-founder and CEO of Tollbit, adverts are being served in OpenAI’s agent mode in ChatGPT, which may full related duties to an AI headless browser.
“If that is triggering an precise advert serving name… multiply that by one million and it will probably principally be… unbelievable scale advert fraud,” the enterprise publishing exec stated, who added that this has the potential to drive down CPMs.
“The primary [next] step might be data,” the exec stated. But it surely wasn’t just like the exec deliberate to deliver this up on the subsequent advertiser shopper assembly. “As a writer that depends on promoting to maintain the lights on, we are able to’t be going out and beating the drum saying, ‘That is unhealthy.’ We’ll allow them to come to us on that one.”
The price of loading webpages in response to AI agent requests can add up for publishers too, as a result of they must pay for higher cybersecurity instruments, Panigrahi stated.
Jesse Dwyer, head of communication at Perplexity, stated the prices have been far better for Perplexity. “This does create some actual value concerns for publishers and AI firms to sort out collectively. When a consumer sends their Comet assistant to a web site, it prices Perplexity 350X to 2300X greater than the writer’s value to serve that consumer a single web page view. The brand new prices of giving folks a greater web are proof that customers have been annoyed by the previous web. So publishers and AI firms are collectively on the forefront of making a greater, more healthy web,” Dwyer stated.
How can publishers detect headless searching?
There are some alerts publishers can search for that point out they’re receiving visitors from AI headless browsers, Panigrahi stated.
If the request isn’t rendering JavaScript, it’s possible not from a human – as a result of most individuals have JavaScript of their browsers, Panigrahi stated. If the IP tackle is coming from a knowledge heart, that may also be a purple flag. The issue is that many of those requests come from residential IP addresses.
The digital publishing exec stated there are a number of indicators they’re in search of now, comparable to how many individuals are searching its websites utilizing the Comet browser. However that also doesn’t make it clear if the searching is being performed by an AI agent or human.
“In the meanwhile, there’s no flag getting handed by means of from the browser to the web site that tells you whether or not the browser is being human-controlled or AI-controlled,” the exec stated.
