As generative AI reshapes how content material is surfaced, scraped, summarized and monetized, new vocabulary to explain these processes is cropping up day by day within the media business.
In technique decks, licensing talks, analysis studies and closed-door conferences, publishers are navigating a flood of latest phrases, together with “zero-click visitors,” “RAG brokers,” and “Mannequin Content material Protocol.”
Right here’s a jargon buster to assist make sense of the rising vocabulary of AI-media economics, what these phrases imply, and why they matter now.
Pay-per-crawl
What it feels like: A toddler health class
What it means: An rising concept (and negotiating tactic) the place AI firms would pay publishers each time a bot crawls their web site to ingest content material. It’s not a lump sum like a licensing deal, however it might cost for each digital chunk. So any time an LLM crawls and recrawls a web site, it’d pay a pre-agreed quantity to the writer. It stays to be seen whether or not it scales.
Pay-per-query
What it feels like: A really costly Google behavior
What it means: This appears to be the favored AI cost idea amongst some up to now. It’s a proposed mannequin the place publishers receives a commission each time their content material powers an AI-generated reply. So not when it’s crawled, however when it’s used. (The argument is one crawl may feed a number of question outcomes, which publishers wouldn’t receives a commission for.)
AI agent
What it feels like: A robotic assistant with its personal agenda
What it means: One of many scorching phrases of the yr, these bots use generative AI expertise to finish duties on behalf of a person. They’ll act like digital assistants. It’s a extra refined AI bot (generally used interchangeably, however “bots” sometimes carry out less complicated duties in a conversational format — typically with scripted responses to prompts) that may act independently. They’ll purchase groceries and guide restaurant reservations on the behest of a person. In accordance with Google’s definition, bots are “reactive” whereas AI brokers are “proactive.”
AI person agent
What it feels like: An avatar in a digital world
What it means: This time period can be typically used interchangeably with “AI agent,” however AI person brokers particularly work together with on-line content material on behalf of a person. For instance, a person may ask an AI system, “What’s Google Zero?” The system would inform the AI person agent to go retrieve that data, it might crawl websites like Digiday’s to get that data, after which the AI system would generate a response to the person primarily based on that information. Some examples of the highest AI person brokers embrace: GPTBot, Google-Prolonged, ClaudeBot, Anthropic-AI, PerplexityBot, Bytespider and ChatGPT-Consumer.
RAG agent
What it feels like: A spy who moonlights as a reality checker
What it means: There are a lot of several types of bots crawling the online for data, and that is one in every of them. Brief for Retrieval Augmented Era — a technical course of for extracting data from content material to present to an LLM and enhance the standard of its output — these AI brokers search the online and retrieve up-to-date, current data to reply to customers’ prompts in AI chatbots like ChatGPT. Generated responses typically embrace citations to unique sources of data. The method is usually referred to as “inference,” which refers back to the time when an AI mannequin solutions a query.
Coaching information crawler
What it feels like: A content-hungry robotic that by no means sleeps.
What it means: Not like a RAG agent, this sort of AI bot collects information to coach LLMs, similar to Meta’s Llama or OpenAI’s GPT. These bots scour the online and obtain content material to construct massive datasets to show an LLM methods to generate responses to questions.
Some tech firms separate the online crawlers they use to coach their AI fashions from the bots that scrape the online for real-time data, in order that publishers can block these bots. However some, like Google, don’t.
For instance, Google-Prolonged is Google’s bot that crawls the web to coach its AI fashions. This may be blocked with robots.txt (see beneath). Googlebot, in the meantime, crawls websites for search indexing and to energy real-time leads to its AI chatbot Gemini and AI Overviews, Google’s AI-generated summaries in its search outcomes. It’s grow to be an actual sticking level for publishers, who can’t block the bot and defend their content material from getting used for coaching Google’s AI merchandise, with out eradicating themselves from search outcomes.
RAG bot scraping overtook the scraping exercise of coaching bots this yr, in accordance with a report by TollBit, an information market for publishers and AI firms.
“RAG scrapes are going to be way more quite a few as AI brokers proceed to extend, whereas scrapes for coaching can be far fewer and doubtlessly extra helpful. Additionally it is more durable to cost content material for coaching, whereas RAG utilization is less complicated to cost and might be dynamic as a result of it’s fixed, recurring entry,” stated Toshit Panigrahi, TollBit co-founder and CEO.
Mannequin Context Protocol
What it feels like: A sci-fi rulebook
What it means: This framework, created by AI firm Anthropic, is a option to index content material and plug it into an LLM in order that it may instantly pull data from a dataset. If RAG is sort of a bridge from an AI system to on-line content material, MCP is like an AI system plugged instantly into that content material’s outlet.
For instance, a person may take all of their information subscriptions and plug them right into a chatbot in order that they’ll entry all of their information content material in that interface, slightly than going to a writer’s web site to get that data, in accordance with Nicholas Diakopoulos, computational journalism professor at Northwestern College.
Which will sound terrifying to a writer already seeing onsite visitors decline – however this may very well be one other platform for content material distribution, and even monetization, Diakopoulos stated.
“As an finish person… I can get the entire worth of the content material that I subscribe to by means of no matter interface I would like. And I believe that sort of user-oriented pondering could have some worth,” he stated. “Is there at the moment a marketplace for that? Not that I can see, however there may very well be one, and I’d assume that information organizations is likely to be on the forefront of inventing that market.”
Robots.txt
What it feels like: Directions for well-behaved machines
What it means: This humble textual content file tells bots which elements of our web site they’re allowed to crawl. In principle, it’s like a “preserve out” signal for AI scrapers. However in observe, most LLMs both ignore it, misinterpret it, or fully flout it. Some will even create a brand new crawler beneath a brand new title if their unique one has been informed to cease crawling. Publishers’ bane.
Google Zero
What it feels like: A brand new dystopian film the place nobody wins
What it means: The second when Google is not the gateway to the web and turns into an AI-powered reply engine to customers’ queries. This new period is being pushed by AI Overviews. It might be the top of a key visitors channel to publishers.
Crawl-to-referral ratio
What it feels like: A bizarre math class
What it means: This refers back to the ratio of webpage crawls to webpage referrals, by platform. Cloudflare tracks this, with information displaying that AI firms’ crawl-to-referral ratio is huge in comparison with Google’s. (For instance, in June 2025, Google crawls web sites about 14 instances for each referral, in comparison with OpenAI’s crawl-to-referral ratio of 1,700:1). In different phrases, bots are scraping publishers’ websites method, far more than they’re referring actual individuals again to these websites.
AI browser
What it feels like: The browser that is aware of you higher than your greatest pal
What it means: Browsers have come a great distance for the reason that OG Web Explorer. These new platforms powered by AI expertise don’t simply present hyperlinks to websites and supply a gateway to the online. They’ll carry out duties, perceive pure language, ship emails, generate briefs and guide conferences. Notable AI browsers embrace Dia (from Browser Firm of New York), Comet (from Perplexity) and OpenAI’s browser, reportedly coming quickly.
LLM Content material Ingest API
What it feels like: A smoothie for robots
What it means: The IAB Tech Lab’s initiative to create a market commonplace for publisher-consented bot scraping, which attributes and compensates publishers when their content material is scraped. It helps each pay-per-crawl and pay-per-query.
Crimson button
What it feels like: One thing from a spy film
What it means: Cloudflare’s “pink button” (launched on July 1) is principally a panic swap for publishers to dam AI crawlers. It’s grow to be affectionately identified amongst publishers because the pink button.
NLWeb
What it feels like: A futuristic internet constructed by aliens
What it means: Microsoft’s pure language internet protocol that debuted in Could. The aim behind the NLWeb mission is to make it simpler for web site house owners so as to add “conversational interfaces” (like chatbots) to their websites. It could additionally assist make web sites simpler to entry by AI brokers that assist MCP. Microsoft thinks this would be the new model of HTML. It may very well be the framework for extra bot visitors monetization fashions.
A2A
What it feels like: A kind of battery
What it means: A Google protocol referred to as Agent2Agent, introduced in April. It’s aimed toward creating a typical language to assist AI brokers which are constructed on totally different frameworks to “speak” to one another, and create a extra interoperable system. Like NLWeb, it additionally helps MPC. It was donated to the non-profit Linux Basis in June.
