This week’s Media Briefing seems at why some publishers that after blocked all AI bots are reconsidering their stance, as rising visitors from AI platforms is pushing them to weigh potential monetization alternatives towards the dangers of unrestricted entry.
Publishers reassess their block-all-AI-bots stance
Some publishers are reassessing their block-all-AI-bots stance.
The controversy over whether or not to dam AI bots has been simmering with publishers for greater than a yr, as AI crawlers proliferated and publishers weighed visitors, licensing leverage, and authorized danger. There may be nonetheless no clear consensus, however 4 publishing execs whose corporations have beforehand taken a tough line on blocking bots advised Digiday that shifts in how AI engines floor and route their content material are forcing them to query whether or not blocking is the precise name. They’re now questioning whether or not they’re sacrificing the power to monetize visitors that might be pushed to them through AI engines.
Execs advised Digiday that their methods should be extra nuanced in comparison with a yr or so in the past, when AI bot visitors was on the rise. Particularly after (not less than one exec mentioned) they’re seeing extra visitors from AI platforms like Perplexity, Anthropic and ChatGPT, although they didn’t present particular figures.
Will Allen, vp of product at Cloudflare, mentioned its information exhibits a steady enhance in AI bot visitors to web sites.
“We’re seeing broadly — and likewise what we’re listening to anecdotally — there’s progress [in AI bot traffic], and it comes from want in content material for coaching in addition to content material for inference and search,” Allen mentioned.
Anecdotally, Allen mentioned he nonetheless sees publishers take an aggressive method to blocking AI bots. Publishers want to know which bots are accessing their websites, how ceaselessly, and what the worth trade is (resembling referral visitors or conversions), Allen mentioned. Then publishers want to determine whether or not they’re getting worth from these visits, and decide which explicit crawlers they wish to give entry to their content material.
This yr, new instruments from corporations like Cloudflare, Tollbit, Fastly and DataDome imply publishers have a greater sense of which AI bots are coming to their websites, and the way typically.
In a DataDome report out final month, LLM crawler visitors quadrupled throughout the cybersecurity firm’s buyer base in 2025, rising from 2.6% of verified bot visitors in January to over 10.1% by August. DataDome detected almost 1.7 billion requests from OpenAI crawlers in a single month.
Some writer web optimization groups have pushed again internally on bot blocking, a subject that surfaced at Digiday’s Publishing Summit city corridor in Miami, Fla. in September.
“We began to see all this elevated referral visitors coming from LLMs, after which our web optimization staff mentioned, ‘Please don’t block these guys,’” mentioned one publishing exec on the DPS city corridor. “We block bot visitors and now we’re involved that that’s hurting us… Are we dropping out on visitors we might be getting in that point once we’re dropping visitors? Possibly we have to flip this off… We are able to assume most visitors [to our sites] goes to be coming from LLMs within the subsequent few years, perhaps sooner.”
That exec mentioned their firm’s AI committee analyze its bot visitors quarterly to see if it’s essential to reevaluate its technique. At present, their firm redirects AI coaching bots by means of Tollbit, however lets AI search bots by means of. AI coaching bots crawl the net for information to feed into LLMs, whereas AI search bots (additionally known as RAG AI bots, or brokers), retrieve info in actual time in response to person prompts in AI merchandise like ChatGPT by looking the net — and sometimes embody hyperlinks or citations to the unique sources, resembling publishers’ websites. AI corporations should self-identify these bots for publishers to know what they’re (for instance, OpenAI’s RAG AI bot is named “ChatGPT-Person” whereas “GPTBot” is its coaching information bot).
The writer’s website receives weekly AI bot visitors “within the thousands and thousands,” the exec mentioned. Blocking that visitors could imply not having their content material surfaced in AI merchandise like ChatGPT, probably lacking the chance to get in entrance of the AI chatbot’s 800 million weekly customers, they mentioned.
“No selections have been made but [on whether to soften its former bot-blocking stances],” mentioned one other publishing exec who spoke to Digiday beneath the situation of anonymity. “[We] leaned towards blocking AI bots as a result of the worth trade wasn’t clear [and] content material was being ingested with out compensation or attribution. In the present day, it’s turn out to be a extra nuanced dialogue. We’re weighing the dangers of free use towards potential alternatives round visibility, attribution and licensing… is it time to open that as much as not forsake potential visitors when web optimization visitors isn’t enhancing?”
A 3rd nameless publishing exec mentioned their firm continues to be attempting to find out the broader implications, and the way it could “really work” to permit some AI bots in and never others. Their firm has blocked all AI bots for the previous two years.
“The dialog is on the sting now. What’s it that we would like the LLMs to learn about? Appears self-defeating to dam 100% of all the things when there are issues we would like the world to [see what we publish],” they mentioned. “A lot of our considering’s evolution has needed to do with realizing that bucketing all content material as ‘content material writ giant’ is a bit simplistic, perhaps knee-jerk. Not all of a writer’s content material must be valued the identical.”
It’s additionally simply a part of the pure evolution of getting extra comfy with AI, the third publishing exec mentioned. As their firm makes use of the know-how extra internally, different execs on the firm are gaining a greater understanding of the worth trade. “Discussions of ‘instruments in trade for content material’ are not non-starters,” they mentioned.
On the Prebid Summit on Tuesday, Paul Bannister, chief technique officer at Raptive, spoke about this dilemma throughout an onstage session: “A lot of the LLMs have a number of bots, and you need to perceive which of them you wish to block wherein eventualities. And it’s not as simple [as] ‘all the time block.’ It’s determining the nuance of, what do you want to do in what state of affairs, and what issues would possibly you have the ability to earn money from sooner or later, and what issues are unlikely to ever… receives a commission for? And determine that hole.”
What we’ve heard
“I’d outline us as a free rider. Like, I’m not the man that’s going to go off and sue OpenAI. We’re too small, and it’s too time consuming… You wish to be aligned with all these guys [like Cloudflare and Tollbit], so that you’re getting near it.”
—An nameless CEO at a media firm on their AI bot blocking technique.
Regulator CMA sharpens deal with AI Overviews and writer visitors
U.Ok. regulator the Competitors and Markets Authority (CMA) has formally put Google’s AI search options that have an effect on writer visitors — AI Overviews, AI Mode, and Uncover — beneath the microscope.
The CMA formally designated Google as having Strategic Market Standing on Oct. 10. In a nutshell, it’s a procedural win with sensible upside. For publishers, it opens the door for the CMA with the ability to push for minimal attribution requirements inside AI solutions, hyperlink prominence guidelines, and maybe even testing obligations earlier than widescale UI modifications. Backside line: it could imply regulatory recognition that AI summaries siphon clicks.
With any regulatory modifications, there are some caveats, one being how lengthy it takes to get any treatments into play, but it surely’s an important step in the precise route, in accordance with the Skilled Publishers’ Affiliation (PPA). Additionally, Gemini wasn’t included inside the scope, that means the CMA might want to guard towards scope-shift workarounds.
The PPA, which counts Hearst UK, Condé Nast and Rapid Media amongst its members, has submitted in-depth proof to the CMA that paperwork a direct correlation between writer citations in Google AI Overviews growing, and click-throughs dropping — what the PPA known as “the crocodile phenomenon” in its proof to the CMA.
“The large downside we wished to resolve is that, as issues at present stand, you’re seeing claims from Google saying that the AI merchandise are literally good for publishers, and enhance their visitors, however not publishing any of the info that might show that,” mentioned Eilidh Wilson, head of coverage and public affairs on the PPA.
This crocodile pattern that the PPA has recognized in its proof exhibits search case examples inside its membership, the place publishers are seeing a rise in appearances in search and a lower in visitors, she careworn. “So the impressions and rankings go up, demonstrating that their content material is extra fascinating to the [AI] product, however the visitors goes down, so the precise reward for the writer is diminishing and in some instances non-existent,” she advised Digiday.
Here’s what the PPA is lobbying the CMA exhausting for about Google’s stage of transparency:
- Publish separate crawl stories that clearly distinguish indexing crawls from crawls used to generate AI Overviews, AI Mode, or any future AI abstract/aggregation options.
- Add Search Console visibility displaying when and the place a writer’s content material is surfaced inside AI-powered SERP components (AI Overviews, AI Mode) and AI Overviews inside Uncover.
- Present clear acquisition supply labeling in Google Analytics so AI-driven referrals are identifiable.
- Mandate clear attribution and clickable supply hyperlinks every time writer content material powers AI Overviews, AI Mode, or Uncover.
In November, the CMA will publish a draft algorithm for the way Google ought to behave after which ask publishers and Google for suggestions. Publishers aren’t asking to kill AI snippets, famous Wilson, reasonably that they need the AI options to work for them too, with clear credit score and hyperlinks that truly ship readers to their websites. — Jessica Davies
Numbers to know
$500 million: The quantity The Los Angeles Instances needs to boost forward of its plans to go public by 2027.
$130 million: The quantity Bustle Digital Group is anticipating to earn in income this yr.
226: The variety of individuals laid off at Folks Inc. (previously Dotdash Meredith), representing 6% of its workforce.
150: The variety of individuals anticipated to be let go at NBC Information, which has about 2,000 workers.
What we’ve lined
Overheard on the 2025 Prebid Summit
- This yr’s version of the annual, publisher-focused advert tech convention had a selected sense of urgency given Google’s AI Overviews looming over the media enterprise.
- A number of the business’s OGs have been available to make clear controversial strikes, and to tease soon-to-be revealed options.
Get the important thing takeaways from the occasion right here.
Why 1440 is evolving from a e-newsletter firm to a vacation spot of explainers
- Each day electronic mail e-newsletter 1440 is evolving into what it’s calling a “information collective,” making a library of explainers by itself web site on subjects that its readers wish to study extra about, as the way in which readers discover and eat info is altering.
- There at the moment are over 200 Matters pages throughout classes like enterprise & finance, science & know-how, well being & drugs, world historical past and society & tradition. The purpose is to get to 10,000 subjects.
Learn extra about 1440’s repositioning right here.
Writer alliance Ozone makes a bigger play for U.S. advertisers
- The writer alliance Ozone is on a progress tear within the U.S., bringing on corporations like Wall Road Journal, New York Publish, the BBC (US) and CNN.
- What started as a pure stock play in 2018 has since advanced right into a broader audience-connection platform that may map viewers content material consumption patterns throughout the publishers to offer a stitched-together view of readers’ behavioral patterns. So far, the mannequin has pushed greater than £400 million ($532 million) in incremental spend again into the U.Ok. publishing sector.
Learn extra about Ozone’s U.S. push right here.
Podcast business juggles utilizing AI instruments with sustaining medium’s intimacy
- Whereas the podcast medium prides itself on its intimate relationship with its listeners and the private connection between host and viewers, AI instruments are more and more used to translate and edit podcast content material.
- Podcasters are grappling with placing a steadiness between defending the character of podcasts and evolving the house for the AI period.
Be taught extra about how podcasters are weighing the usage of AI know-how right here.
The Impartial maps four-pillar progress plan for the AI period
- The Impartial is leaning on 4 progress pillars — its expertise and IP-led verticals, U.S. growth, e-commerce and inner AI merchandise — to resist the zero-click period and the knock-on impact which will have on show advert revenues.
- Every pillar performs a unique function, from driving new income streams to testing how AI could be turned from a menace to a chance.
Learn the dialog with The Impartial’s CEO Christian Broughton right here.
What we’re studying
Plans for NBC Information revealed
Cesar Conde, chairman of the NBCUniversal Information Group, shared in an inner memo what the way forward for NBC Information will appear like, together with the launch of a brand new subscription product later this yr and increasing its sports activities hub, Adweek reported.
New York Instances beefs up AI initiatives staff
The New York Instances named Mukul Devichand editor of the AI Initiatives staff to steer two new user-facing initiatives centered on audio and search, in accordance with MediaPost. Dylan Freedman was promoted to AI initiatives editor to assist reporters with analysis and investigations.
CNN checks new creator unit
CNN’s worldwide division is creating a brand new unit known as CNN Creators with a weekly present and multiplatform content material on AI, tech, artwork, tradition, sport and social traits, in accordance with The Hollywood Reporter.
Perplexity not taking new promoting offers
Perplexity just isn’t taking up new advertisers (writer individuals in its income share mannequin get a lower of advert income), and adverts won’t be a part of the monetization technique for its AI browser Comet, Adweek reported.
Gannett Media president on Perplexity partnership
In an op-ed, Kristin Roberts, president of Gannett Media, defined that the native media community partnered with Perplexity to floor its native information protection within the AI platform and to generate new income.
