How’s your human-in-the loop strategy working out when it comes to implementing AI? In my conversations with editorial leaders, I repeatedly hear about how, in overworked newsrooms, overseeing AI can turn into another task that journalists have little appetite or bandwidth for.

Of course, accuracy and audience trust are not negotiable for any of us in the news business. So, I was intrigued to hear about how Jessica Davis at USA Today Network, the largest network of newspapers in the United States, is using AI strategically to scale AI adoption — without scaling risk. 

“When we consider the future of work, especially for media organisations, reviewing every output puts a ceiling on what's possible to fully embrace the promise of agentic AI. Why I dug into this problem is that a human in the loop, while important, puts a cap on how fast our industry can innovate at scale,” Davis said.

So, Davis said, she took a step back and asked: “How might we create a new operating model from human-in-the-loop on every output, to human-in-the-loop brought in strategically when most needed?”

Her solution — one of the capstone projects from the City University of New York’s Executive Program in News Innovation and Leadership programme — lies in AI evaluations, which assess whether AI is doing its job correctly, with editorial judgment embedded in them.

She used the example of a Freedom of Information request (or public records request) bot that the USA Today team built using Microsoft Copilot.

First, they had to define the task: Journalists wanted to file hundreds of record requests quickly. Then they had to define what success looked like: Accuracy was important, as were responses.

“At first, the agent hallucinated or made up e-mail addresses,” she said.

They also had to assign a tester — ideally, a journalist with subject matter expertise. The team worked closely with journalists on generating requests and improving the accuracy rate.

The final step was to iterate and refine their process. At this point, they created a supervising agent, working with a journalist, an LLM, and a technical product manager to test hundreds of requests at a time.

Accuracy was in the high 90% range “and we were able to go from taking months to deploy to production to shipping multiple features within days,” she said.

“What we learned from this is, we let the LLM do the heavy lifting and give editors something to react to. This is faster and more accurate than reviewing every output. Journalists are so busy — they don’t have time to fill out long forms. We heard this loud and clear, and this actually brought product and editorial together. The AI evals brought people together to collaborate, unified with a shared definition of success.”

She pointed out that software companies such as Microsoft, Anthropic, and Google also use AI evaluations.

“What great AI evals do is that they create data, and that removes black boxes, and great data provides a feedback loop for continuous improvement and meeting of actual workflow needs.”

USA Today is now collaborating with Thomson Reuters on AI evals for both newsrooms.

“This is the new governance model through accountability of AI that works for us. This shifts human-in-the-loop to where it is needed most,” Davis said.

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

Banner photo: Adobe stock image by Photo Agency.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .