But what happens when it’s just not possible to keep a human in the loop?

As the use of AI ramps up within organizations and as businesses delegate more tasks to agentic AI, it’s a challenge that many corporate leaders are starting to grapple with.  

“We’re seeing a material increase in the speed with which we can create things,” Zach Maybury, chief technology officer at online sports betting platform DraftKings said during a panel discussion at Fortune’s flagship technology conference, Brainstorm Tech, this month.

Maybury said his company already deals with trillions of transactions and highly distributed workloads. Once you introduce agentic AI into the mix—with AI agents talking directly to other AI agents—the volume and complexity of the operation is far too vast for traditional approaches.

“I can’t insert humans into those loops,” Maybury said. “We will never have enough humans to insert in all those loops.”

Maybury was just one of several business leaders at Brainstorm Tech who discussed the challenges of managing AI in mission-critical situations.

“In high-stakes environments like health care, it’s not taking the wrong tee-shirt size if you’re a retailer, it’s a life on the other side of it,” said Salesforce chief customer and commercial officer LaShonda Anderson-Williams.

While there is no one-size fits all solution to these challenges, many of the panelists described techniques and frameworks that have proven successful for them. 

Anderson-Williams said taking a clear-eyed look at AI use cases, and understanding what outcome you’re ultimately aiming for, is critical.

Equally important is nailing down the proper governance framework—that is, a clear policy and set of rules that stipulate where and how the AI is allowed to operate, how it’s designed, and who is responsible for various parts of the process. As companies expand their AI and agentic use from small-scale experiments to broad rollouts with high stakes, an up-to-date governance framework is indispensable. 

“A lot of people just ran and bought a lot of different tools and technologies and just bolted them on, and there wasn’t any governance on how the tech was applied,” said Anderson-Williams.

DraftKing’s Maybury said that having a solid AI governance foundation in place provides important safeguards and helps mitigate risk. That might mean taking a hard look at existing processes and making changes, revisions, and expansions to the old governance rules.

“It’s got to be governance that can scale,” he said. 

Anthony Moisant, Indeed’s chief information and security officer, echoed Maybury’s comments about the challenge of having humans in the loop throughout a high-volume job listings service used by 645 million job seekers and 3.5 million employers. He suggests constant testing of processes involving AI to gauge how the results compare to the desired outcomes.

It’s also important to consider the type of situation where AI is being deployed, said Diya Jolly, chief product and technology officer at accounting software firm Xero. Is it something that requires judgment, or something with a clear answer?

“If your outcome is deterministic, then you can probably let the agent go pretty far,” Jolly said, noting that the outcomes in those situations can easily be tested and measured against the desired result. But, she said, “when you have judgement within the decision, that is when it becomes really hard to take the human out of the loop.”