Though AI is altering the media, how a lot it’s altering journalism is unclear. Most editorial insurance policies forbid utilizing AI to assist write tales, and journalists sometimes don’t need the assistance anyway. However when consulting with editorial groups, I usually level out that, even if you happen to by no means publish a single phrase of AI-generated textual content, it nonetheless has so much to supply as a analysis assistant.
Nicely, that assertion could be a bit extra questionable now that the Columbia Journalism Evaluate has gone and printed its personal research about how AI instruments carried out in that position for some particular journalistic use circumstances. The end result, based on CJR: AI could be a surprisingly uninformed researcher, and it may not even be a fantastic summarizer, at the very least in some circumstances.
Let me stress: CJR examined AI fashions in journalistic use circumstances, not generic ones. For summarization particularly, the instruments—together with ChatGPT, Claude, Perplexity, and Gemini—had been requested to summarize transcripts and minutes from native authorities conferences, not articles or PowerPoints. So among the outcomes might go in opposition to instinct, however I additionally assume it makes them way more helpful: For synthetic intelligence to be the pressure for office transformation because it’s usually hyped to be, it wants to provide useful output in workplace-specific use circumstances.
{“blockType”:”creator-network-promo”,”information”:{“mediaUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/mediacopilot-logo-ss.png”,”headline”:”Media CoPilot”,”description”:”Need extra about how AI is altering media? By no means miss an replace from Pete Pachal by signing up for Media CoPilot. To study extra go to mediacopilot.substack.com”,”substackDomain”:”https://mediacopilot.substack.com/”,”colorTheme”:”blue”,”redirectUrl”:””}}
The CJR report reveals some fascinating issues about these use circumstances and the way journalists method AI usually. However largely it reveals how badly we’d like extra of this: systemic testing of AI that goes past the advert hoc experimentation that has too lengthy been the default for a lot of organizations. If the research reveals nothing else, it’s that you simply don’t should be an engineer or a product designer to evaluate how properly AI might help in your job.
Placing AI to the newsroom check
To check AI’s summarization skills, the evaluators—which included lecturers, journalists, and analysis assistants—created a number of prompts to create brief and lengthy summaries from every software, then ran them a number of occasions. A weak point of the report is that it doesn’t reveal the outputs so we are able to see for ourselves how properly it did. However it does say it quantified factual errors to judge accuracy, evaluating them with human-written summaries.
With out seeing the outputs, it’s laborious to know learn how to enhance the prompts to get higher outcomes. The research says it bought good outcomes for brief (200-word) summaries however noticed inaccuracies and missed info in longer ones. One stunning end result was that the only immediate, “Give me a brief abstract of this doc,” produced probably the most persistently good outcomes, however just for brief summaries.
The research additionally checked out analysis instruments, particularly for science reporting. I really like the specificity right here: The CJR researchers had been very specific concerning the use case: giving the software a paper after which asking it to carry out a literature assessment (discovering associated papers, citing them, and extracting the general consensus). Additionally they selected their targets intentionally, evaluating AI-powered analysis providers like Consensus and Semantic Scholar as an alternative of the standard common chatbots.
On this, the outcomes had been arguably even worse. The instruments sometimes would discover and cite papers that had been utterly totally different from what a human picked for a manually created literature assessment, and even totally different from the opposite instruments. And once they ran the identical prompts a number of days later, the outcomes would change once more.
Getting nearer to the metallic
I believe the research is instructive past the easy takeaways, comparable to utilizing AI just for brief summaries and considering twice earlier than utilizing AI analysis apps for literature assessment.
- Immediate engineering issues: I get that the three totally different prompts for summaries had been most likely designed to simulate informal use—the sorts of pure language textual content a busy journalist would possibly sprint off. And perhaps AI ought to in the end produce good outcomes while you do this. However for out-of-the-box instruments (which is what they used), I’d suggest extra considerate prompting.
This doesn’t need to be a giant train. Merely going over your immediate to make obscure language (“brief abstract”) extra exact (“200-word abstract”) would assist. The researchers did ask for extra element in two of the three prompts, however the research criticizes the longer summaries for not being complete when the language within the prompts doesn’t particularly point out comprehensiveness. Asking the AI to verify its personal work generally helps too.
- The app layer struggles: Studying the half concerning the numerous analysis apps not producing good outcomes had me nodding alongside. I don’t need to learn an excessive amount of into this because the research was narrowly targeted on analysis apps with a really particular use case, however I’m at present residing by one thing related whereas experimenting with AI content material platforms for my plans at The Media Copilot. If you use a third-party software, you’re an additional step faraway from the muse mannequin, and also you miss having the pliability of being “nearer to the metallic.”
I believe this factors to a elementary misunderstanding of the so-called “app layer.” Most AI apps will put a veil over system prompts and mannequin pickers within the title of simplification, but it surely isn’t the UX win that many assume it’s. Sure, extra controls would possibly confuse AI newbies, however energy customers need them, and it seems the hole between the 2 teams may not be very giant.
I believe this identical misunderstanding is what stymied the GPT-5 launch. Eradicating the mannequin picker—the place you could possibly choose between GPT-4o, o4-mini, o3, and so on.—appeared like a wise, simplifying concept, but it surely turned out ChatGPT customers had been extra refined than anybody had thought. The common ChatGPT Plus subscriber may not have understood what each mannequin does, however they knew which of them labored for them.
- Iterate, iterate, iterate: The research’s outcomes are useful, however they’re additionally incomplete. Testing outputs from fashions is simply the starting of the method of constructing an AI workflow. When you’ve bought them, you iterate: Regulate your prompts, refine your mannequin alternative, and check out once more. And once more. Producing constant outcomes that save time isn’t one thing you’ll get excellent on the primary strive. When you’ve discovered the suitable mixture of prompting, mannequin, and context, you then’ll have one thing repeatable.
Coming midway
The place does this depart newsrooms? This would possibly sound self-serving since I prepare editorial groups for a residing, however after studying this report, I’m extra satisfied than ever that, regardless of predictions that apps and software program design will summary away prompting, AI literacy nonetheless issues. Getting probably the most out of those instruments means equipping journalists with the abilities they should craft efficient prompts, consider outcomes, and iterate when obligatory.
Additionally, the CJR research is a wonderful template for testing instruments internally. Get a staff collectively (they don’t should be technical), craft prompts methodically, after which consider them—however then iterate. Maintain experimenting. Discover what persistently will get good outcomes—not simply high quality outputs, however a course of that really saves time. Simply doing “vibe checks” gained’t get you very far.
As a result of there may be yet another factor the research is off-target about. When a journalist considers learn how to full a job, the selection normally isn’t between a machine output and a human one. It’s the machine output or nothing in any respect. Some would possibly say that’s decreasing the bar, but it surely’s additionally placing a bar in additional locations. And with some coaching, experimentation, and iteration, elevating it inch by inch.
{“blockType”:”creator-network-promo”,”information”:{“mediaUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/mediacopilot-logo-ss.png”,”headline”:”Media CoPilot”,”description”:”Need extra about how AI is altering media? By no means miss an replace from Pete Pachal by signing up for Media CoPilot. To study extra go to mediacopilot.substack.com”,”substackDomain”:”https://mediacopilot.substack.com/”,”colorTheme”:”blue”,”redirectUrl”:””}}
