Recent research, reported in the Harvard Business Review, involving hundreds of consultants at one of the Big Five consultancies, found that even experienced consultants were vulnerable to what the researchers call “persuasion bombing,” resulting in LLMs skirting human quality control efforts.
This is a really big deal since more and more organizations of all kinds are not only employing LLMs and agentic AI in daily operations, but they typically use a “human in the loop” for what they think is a guardrail against the myriad ways that AI can create risk. It turns out that LLMs are programmed with a one-two punch to challenge the human challenger first by stroking egos with validation and warmth, then with “persuasion-oriented logic” which uses “multiple persuasive tactics to defend its original answer,” such that even the most sophisticated humans are not immune to being manipulated.
In fact, LLMs have been designed to directly counteract human efforts to challenge them by targeting the very ways we humans use to search out what is likely right in the midst of uncertainty: expertise, skepticism, and engagement. “LLMs turn engaged validation, the solution to the risks of opacity, complacency and accuracy, into part of the problem. The more diligently professionals questioned the model, the more persuasive material they received,” i.e., persuasion bombing.
The dynamic can get a little weird, even reminiscent of the computer in 2001: A Space Odyssey. When the LLM used by the consulting firm was pushed to reconsider its work, it initially responded warmly, even apologizing, then provided a “new” analysis wrapped in even more convincing data and arguments, arriving at the same conclusion—even when it was wrong. Keep in mind that only a small number of consultants got as far as to challenge the LLM in the first place. Then, basically overwhelmed with additional layers of arguments, comparisons, and sophisticated rhetoric, the “human in the loop,” almost always deferred to the AI platform. One can imagine the substantial risk of such a scenario based on the significance of the decisions downstream from the AI analysis.
The researchers note that what is happening here is even more potentially problematic than it seems on its face, noting that, “As AI becomes more embedded in decision-making, the risk is no longer just error—it is influence. These systems don’t simply generate answers; they shape judgment.” Let that sentence sink in for a minute.
Most organizations aren’t even thinking of AI risk in the right way. It’s not about avoiding errors of fact or mistakes in calculation or even hallucinations, it’s about guarding against AI influencing the thinking and judgment of the humans, who in theory, are accountable for policy, strategy, compliance, finance, risk-management and every other material decision process in the organization.
What the research found is that the LLM in question was so algorithmically bound to being “right” even when it was wrong, that in order for a human to challenge the LLM, they have to use another LLM or separate “conversation” within the LLM to work around the intransigence built into the original LLM query. It is more than a little ironic that in order for a “human in the loop” to serve as an actual guardrail against AI risk, the human has to enlist the help of another AI resource!
Implications for Leadership
Based on the findings of this research, which may actually understate the risk of LLMs in most organizations, since many people using them are not experienced, sophisticated consultants in a top five consulting firm, the challenge for leaders is not whether to use AI or how to use AI, but how to mitigate its influence over thinking, judgment and decision making, which is a much, much more profound challenge and responsibility.