If you ask an artificial intelligence whether an answer is correct, you’ll almost always get a confident affirmation. AI doesn’t pause to doubt or say, “Maybe I should check”: it returns an answer just as effortlessly as it might produce a completely different one.
In 2023, two New York lawyers found this out the hard way. They had used ChatGPT to draft a legal document and cited six court precedents that the tool had completely fabricated. When their counterparts on the opposing side couldn’t find those rulings, the lawyers asked ChatGPT directly whether the cases were real—and the tool confirmed they were. The federal judge sanctioned the attorneys, pointing out that no one had verified the citations with a reliable third-party source before filing them with the court. It wasn’t a matter of using the wrong tool: anyone who had opened Westlaw or even just Google Scholar to cross-check the information would have realized it within two minutes.
The Real Bottleneck
In companies that are adopting GenAI, the conversation almost always centers on technical aspects: how to write an effective prompt, which features to enable, or how to integrate the tool into workflows.
However, there is a more nuanced aspect missing: knowing when an output is trustworthy and when it should be questioned. This concept has a specific name; in the literature, it is called “calibrated trust,” and it is not a recent development.
The reference model is the one proposed in 2004 by researchers John D. Lee and Katrina See, who were the first to identify two opposing concepts in trust toward an automated system: underutilization, when reliable assistance is ignored simply because it is artificial, and overutilization, when an output is accepted without question even when the system’s actual reliability does not justify it. Calibrated trust, on the other hand, is the kind of trust that keeps pace with the tool’s actual capabilities, not with the confidence with which it presents them.
The Skills You Really Need
Learning when to trust AI output—and when to question it—means constantly training yourself to ask questions, even uncomfortable ones. Because this assessment isn’t just about technology; it involves critical thinking, an understanding of context, and, not least, the honesty to acknowledge one’s own blind spots.
A model can be trained to cite sources or to indicate its own uncertainty, but the final decision on how much to trust it remains a purely human responsibility—one that must be practiced and cultivated.
A study published in early 2025 by Microsoft Research and Carnegie Mellon University—which surveyed knowledge workers who use AIGen tools at least once a week—highlighted precisely this mechanism.
The authors found a clear correlation: the more a professional trusts AI’s ability to perform a task, the less they engage their critical thinking on that task. Those who have more confidence in their own skills, on the other hand, tend to question the output more frequently, verifying, supplementing, and correcting it before considering it final. In other words, “calibrated trust” is not the result of a distrustful attitude toward AI, but rather of the confidence that stems from one’s own experience and expertise. Those who are more familiar with the context and have a deeper mastery of the subject matter are also better able to assess with greater precision when to trust the AI and when to question its responses.
The case mentioned at the beginning is an extreme example of what happens when expertise is completely lacking. But the mechanism that led two lawyers to blindly trust an AI output is the same—on a smaller scale—that leads any of us to copy a piece of data, a number, or a statement generated by AI without verifying it, simply because it sounds plausible.
Why It's Worth Discussing This Now
The organizations that are investing most effectively in generative AI are those that have begun to establish a shared vocabulary among their people for discussing risk and reliability. For example, they know how to distinguish between a low-risk task—where AI can operate on its own—and a high-risk one—where supervision is required. And this distinction arises from practical experience, from mistakes made, and from a corporate culture that allows people to say, “I don’t trust this output,” without feeling at fault compared to those who use it with ease.
AI will never have doubts. It’s up to those who use it to establish the criteria for deciding when to take it seriously and when not to. And that’s a process that starts with people and the workplace culture, even before it starts with the tools.
.png)
