Trusting the machine – Part 1

Aug 19

8 min read

Updated: Aug 20

If you had undeniable proof that a machine made meaningfully fewer mistakes than a human - would you trust it more than a human to make those decisions?

Despite our ever growing reliance, we humans have long struggled ‘trusting the machine’.

The ability to form trust in others, beliefs and systems is the cornerstone of human survival and achievement. From transactional to deeper emotional trust, it is built on real people having stakes in the game, perceived mutual benefit, competence and reliability demonstrated over time. It is deepened with relational intimacy that demonstrates understanding of personal needs, honesty, transparency and care for our personal outcomes. It’s no wonder that this deeply human quality is something we are reluctant to extend to AI.

We are all well accustomed to the vagaries of ‘human error’ and are routinely forced to accept flawed human judgement in medicine, finance, aviation and everyday life. We do not like those mistakes, but we have a language for them. A person was tired or impaired, the situation was unusual or the data was incomplete, the expert just had to make a judgement call. We imagine someone can be sanctioned, punished, retrained or forgiven when required.

Machine mistakes hit different

Machine mistakes feel different. They arrive without a satisfying back story that reassures us this was an acceptable anomaly or that we can assert more safeguards moving forward. An AI system can only offer probabilistic functions as an excuse. That may be statistically honest, but it hardly satisfies our desire for certainty and control.

We are already having to trust the machine but still worry about it at the same time.

A 2025 University of Melbourne and KPMG study covering more than 48,000 people across 47 countries found that 66% use AI regularly and 83% expect it to deliver benefits, yet only 46% are willing to trust AI systems. The 2026 Stanford AI Index tells a similarly conflicted story: optimism about AI rose during 2025, but so did nervousness. Research published in 2025 exploring public trust and blame attribution in human-AI interactions found that people generally attributed more trust and less blame to human operators than AI operators across air traffic control and autonomous vehicle scenarios. Human operators were more morally familiar and therefore often judged more generously.

AI can to some degree meet the emerging trust test as a technical challenge with improved accuracy, reduced bias, model explainability, improved privacy protection, and regulated risk. Beyond this though there is a fundamental human challenge. Systematic mistakes feel infinitely less forgivable than everyday human fallibility. This subjective human weighting is a key barrier to trusting the machines and will be increasingly tested with mass AI adoption throughout many aspects of our lives.

Who ‘cares’?

Some forms of trust are mainly about competence and reliability while other forms contain a hidden relational test – “Do you really care what happens to me?”

Machines themselves do not have a real stake in the consequences. They do not feel shame, guilt, grief, pride or duty. They do not look a family in the eye after a bad outcome or carry regret. An AI system can optimise for your outcome without caring about you. However, a person can care deeply about you and still make a worse decision.

That matters most when a decision touches dignity, identity, suffering or life direction. We may want the decision-maker to be vulnerable to the meaning of the decision, not merely capable of optimising an outcome. Competence and care are different sources of trust, and one does not automatically substitute for the other.

An imperfect world where error becomes measurable

Human error is already built into our most important systems - fallibility is assumed. In fact behavioural science evidence has proved that not only do humans often get things wrong, we are reliably and predictably wrong. The inherent biases influencing our thought processes mean the mistakes we make are often systematic and not random (therefore compounding errors rather than cancelling them out).

We don’t demand perfection from human decisions but we do demand a story. An AI output does not naturally fit that moral vocabulary. Even when we can technically explain the factors influencing a decision, the explanation may still feel empty. It does not tell us what the machine meant, valued, cared about, or what it had to lose.

Trust is the confidence that somebody will perform as promised. Importantly, it is also confidence that, if something goes wrong, the failure can be made meaningful.

Some years ago, my team built a psychographic member segmentation model for a superannuation fund. The model used member data supplemented with detailed primary research on over 10 thousand members. We were able to demonstrate the ability to correctly classify close to 80% of the members into their correct segments based only on BAU data i.e. allowing automated classification in the future without the rich point in time primary research data used to initially derive the segments. This level of accuracy using only proxy data was quite the coup for the data science team at the time.

The reaction from one senior stakeholder as we buoyantly presented our ‘success’ was however revealing. Quantified accuracy became threatening - “So you are telling us you know that the model will be wrong for at least 1 in 5 members”

The funds existing member engagement approach was to communicate with what they believed was the ‘average member’. We found this 'averaging' was masking the true diversity of needs and only reasonably accounted for around 1 in 5 members). They may have in fact sent the wrong message, offered the wrong support or assumed the wrong motivations far more often - possibly for 4 in 5 rather than 1 in 5 members i.e. getting it ‘wrong’ 4 times as much.

However, those mistakes were diffuse with no probability score attached to them and no directly measured accountability. The old system’s human errors were invisible, while the model had made its limitations clear to see and the uncertainty explicit. Once error became countable, it felt accountable for the human signing off. Sometimes distrust of machine learning may really be discomfort with accepting probabilistic uncertainty and having decision error quantified.

Explainability vs reassurance

Classic research on algorithm aversion found that people can lose confidence in an algorithm more quickly than in a human after seeing both make mistakes, even when the algorithm performs better overall. We treat machine error as evidence that the system is defective, not that it is simply imperfect, just like the humans it was trained on.

How many more mistakes would a human have to make than a machine before we feel comfortable passing trust?

Probability is often more honest than narrative, but narrative is often more comforting. Most people are just seeking some reassurance, not the math. Perhaps as the world gets increasingly complex, we are better served learning to live more with explained uncertainty. Making better data-led decisions requires embracing probabilities rather than just seeking narrative reassurances. This does not mean passing trust without questioning, but asking more nuanced questions of the systems we come to trust:

How confident is AI in this decision? What kind of errors does it make in this domain? How does it compare with the human alternative? What happens when it is wrong – both impact and accountability? Can the decision be challenged or vetted? What safeguards exist when the stakes rise?

While less emotionally satisfying than a confident expert with a compelling rationalisation, they may also be more useful for managing decision-making risks.

Who answers when the machine is wrong?

When a human professional makes a serious mistake, there may be an apology, investigation, professional discipline, litigation, retraining or institutional review. These mechanisms are imperfect, but they reassure us the responsibility sits with someone.

With AI, responsibility can diffuse across the model developer, data provider, deployer, interface designer, organisation, regulator, human decision-maker and user. Everyone played a part and there was very likely no malicious intent, which can mean nobody feels fully answerable.

A recent example of machines making contestable decisions in the context of aged care funding (albeit via business rule algorithms rather than AI) was reported by an ABC News investigation. When applying for a reassessment, an elderly funding applicant found that a "classification algorithm" had decided the outcome and the assessing clinician was unable to change it. An appeal to contest the denial of increased funding was subsequently unsuccessful, drawing comparisons to the appeals process of the now discredited Robodebt scheme.

When the somewhat secretive algorithms black box was unpacked by an expert, through documents obtained under Freedom of Information, several potential flaws were exposed around how crucial factors impacted the final score.

Professor Eager who reviewed the case noted "…the onus is on the consumer, the frail older person, they then have to challenge the algorithm… Nobody has to justify why the algorithm gave them that." The government has now promised to legislate a new assessment escalation pathway and confirmed the budget had committed funding to review and refine the algorithm.

It’s not hard to see how many of us could struggle with accepting opaque machine-driven decision making in this context if personally impacted. In contexts where care is crucial to trust, machines might be limited to a calculation role, while people remain responsible for consent and consequence.

The high-stakes test

Imagine an AI-supported diagnostic, surgical or air traffic control system that could reduce serious errors by 10%, 20% or 50%. It would still make mistakes.

No matter how uncomfortable you are trusting a machine - would you accept one mistaken death associated with the machine if its use prevented 5 deaths that would otherwise have occurred through human error? No? What about preventing 10 or a hundred deaths? Is there a number that would change your mind?

We accept human error where we would not accept it from a machine because it remains morally legible. This may become an expensive luxury if it preserves systems with inferior performance simply because their mistakes feel more naturally acceptable. In fact, once we measure improved human outcomes through machine-driven decisions, are we morally obligated to embrace consequentialism and let go of some of our philosophical conundrums with trusting the machine?

AI should not need to be perfect to become trustworthy. It needs to be demonstrably better than the available alternative when used under defined conditions, with safeguards proportionate to the stakes. This is not a call to ‘get with the program’ – it’s a spotlight on a question of trust we will all increasingly face.

In Part 2, we shift from considering the innate human reluctance to ‘trust the machine’ to the practical questions organisations, policymakers and individuals now face.

Selected bibliography

Gillespie, N., Lockey, S., Ward, T., Macdade, A., & Hassed, G. (2025). Trust, attitudes and use of artificial intelligence: A global study 2025. University of Melbourne & KPMG. DOI: 10.26188/28822919.
Stanford Institute for Human-Centered Artificial Intelligence. (2026). The 2026 AI Index Report: Public Opinion. Stanford University.
Mei, P., Cannon, R., Everett, J. A. C., Liu, P., & Awad, E. (2025). Public trust and blame attribution in human-AI interactions: A comparison between air traffic control and vehicle driving. Transportation Research Interdisciplinary Perspectives, 32, 101545. DOI: 10.1016/j.trip.2025.101545.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. DOI: 10.1037/xge0000033.
Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. DOI: 10.1518/hfes.46.1.50_30392.
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709–734. DOI: 10.5465/amr.1995.9508080335.
Reyes, J., Batmaz, A. U., & Kersten-Oertel, M. (2025). Trusting AI: Does uncertainty visualization affect decision-making? Frontiers in Computer Science. DOI: 10.3389/fcomp.2025.1464348.
Elish, M. C. (2019). Moral crumple zones: Cautionary tales in human-robot interaction. Engaging Science, Technology, and Society, 5, 40–60. DOI: 10.17351/ests2019.260.Connolly, A., & Kopel, N. (2026, 24 March). New aged care algorithm under fire as 800 apply for review. ABC Investigations / 7.30.
Inside the aged care algorithm deciding support for older Australians - https://www.abc.net.au/news/2026-08-17/inside-the-black-box-aged-care-algorithm-for-support-at-home/107033970?utm_source=braze&utm_medium=email&utm_campaign=20260816_newsam_newsletter&utm_id=6a822bd96cd5390776ae80d235aadc12
Australian Government Department of Health, Disability and Ageing. (2026). Integrated Assessment Tool (IAT) User Guide and related Support at Home guidance.