top of page
Search

Trusting the machine – Part 2

Aug 19
8 min read

Updated: Aug 26

Recap: In Part 1, we explored why machine mistakes often feel less forgivable than human ones. Human failure can be narrated through intention, pressure, care, negligence and accountability. Machine failure is more visible, measurable and harder to place inside our moral vocabulary. In Part 2 we dive deeper into some practical considerations for building a more robust organisational AI trust architecture.


Trust is not a switch and the threshold to win it should rise with the consequences. We already understand this with people. We trust a friend to recommend a restaurant but not to diagnose a tumour. We trust a pilot within a system of training, maintenance, procedures, air traffic control and regulation. Where there is complexity and stakes, we rarely just trust isolated individuals and typically look for reassurance in qualified roles, systems and safeguards. AI should be no different.


Sometimes we have little choice but to trust. Every time we get on the road, we need to pass trust to complete strangers. We rely on the assumption they have stakes in the game and unsafe driving is just as great a threat to them as ourselves. With the rise of ubiquitous AIoT (artificial intelligence of things), we may have little choice but to trust AI enough to function in a modern world.


To trust or not to trust…is that the question?


While the public debate often assumes a divide between enthusiasts who trust AI and sceptics who do not, human behaviour is a lot messier.


Algorithm aversion means we may abandon a system after one conspicuous mistake, even when it performs better than a person. Automation bias pulls in the opposite direction. We may over-rely on a recommendation because it looks authoritative, arrives quickly or carries institutional approval. Research on human and AI decision-making has documented both overreliance and selective adherence, including when algorithmic advice reinforces existing assumptions.


A chatbot may receive intimate disclosure despite having no genuine duty of care. A strong decision-support system may be rejected after one visible mistake. An employee may ignore one AI warning because it conflicts with intuition, then accept the next recommendation without checking because the deadline is close.


We may reject AI where it could help and obey it where we should question it. So, the danger is more than simply trusting the machine too much or too little – it’s all about context.


The context of trust


Low-stakes trust includes summaries, recommendations, brainstorming and administration. Errors are annoying but usually recoverable.


Delegated trust begins when AI structures decisions around money, work, education, legal information or health. It may not make the final choice, but it shapes which options appear reasonable and which trade-offs receive attention.


High-stakes trust applies where error can cause irreversible harm: diagnosis, autonomous transport, welfare eligibility, criminal justice or critical infrastructure. Performance, oversight and accountability must be demonstrated.


Then there is intimate trust. People ask AI about relationships, identity, career direction, loneliness, whether to have another child, whether they can afford to leave a job or whether their life still makes sense. The risk escalates from factual error to undue influence, dependency and the outsourcing of self-reflection. Perhaps most troubling is how easily we move between these contexts, often without noticing where one ends and the next begins.


Now it’s personal


The most important shift may begin in more personal rather than commercial applications. People already use AI to model financial scenarios, compare life choices, test career moves and think through difficult personal decisions. They may not hand over control, but they allow AI to frame the possibilities and may become influenced far more than they realise.


In May 2026, OpenAI launched a personal finance experience in ChatGPT for US Pro users, later expanding it to Plus users. People can connect financial accounts, see financial information and ask questions grounded in their circumstances. OpenAI states that the service is not a replacement for professional financial advice, but the direction is clear: general-purpose AI is becoming personal context-aware life infrastructure.


People are no longer just asking questions for factual answers they might have previously googled. They are asking for deeply personal insights about their life choices. This becomes more delegated reflection than information retrieval.


At the more extreme end of the scale, the 2026 Stanford AI Index reports that AI companionship remains a minority behaviour, but more than half of people globally express at least some excitement about using AI for companionship. A study conducted by MYMAVINS in 2025, the Real Relationships report, found that 13% of Australian adults using AI tools admit feeling a strong personal connection with them, like with a friend. A further 21% feel a personal connection but only in a light-hearted or casual way.


While still far from universal, it’s no longer science fiction that this is where we could be heading. This is only likely to increase as younger generations are desensitised with higher rates of AI companion exposure from an early age.


As discussed in past MYMAVINS blogs around AI Intimacy, AI can now function as a trusted adviser who has earnt the right to understand you - a coach, a life assistant, a customer advocate, a sounding board and an interpreter and validator of your intimate thoughts.


Most people will not wake up one morning and decide to form a relationship with AI but may notice, one day, that they already rely on it like they do a trusted friend. It may look like trusted impartial advice but when does personalised help become influence? When does influence become manipulation?


This all suggests the next era of personalisation needs a clear trust architecture with transparency forming a core part of the experience. If we rush too quickly into AI intimacy, we may build systems that are persuasive before they are commensurately accountable and trustworthy.


The terms of surrender?


In higher stakes questions of trust, what we really may be negotiating is how much control we are prepared to surrender and on what terms. Therefore, a useful trust architecture should answer seven questions.


1. Visibility

Can people tell when AI is being used, what role it plays and when it is uncertain?


2. Contestability

Can a person challenge, correct or appeal the output in a practical, meaningful way?


3. Proportionality

Is the level of automation appropriate to the stakes? A restaurant recommendation and a welfare decision should not face the same threshold.


4. Accountability

Which organisation or professional owns the decision, investigates harm and provides remedy?


5. Comparative performance

Is the AI better than the real process currently in use, including its delays, inconsistencies, costs and invisible errors?


6. Human dignity

Does the process preserve voice, context and the right to be heard, rather than reducing people to data points?


7. Learning loops

Does the system record failures, detect patterns and improve, or simply repeat mistakes faster and at scale?


For organisations, AI trust is not only an IT or compliance issue but a fundamental part of the customer journey, service design requirements and employee decision-making processes. In the future, these trust proof points may (or at least should) become commercial and operational requirements. The organisations that earn trust will be those that make imperfection manageable rather than claim their systems are flawless.


Practical considerations for organisations


We should want to know that an organisation chose an appropriate use case for AI, tested it in realistic conditions, protected the data, trained the people, created escalation pathways, had a risk mitigation plan and system guardrails, monitored outcomes and accepted responsibility for failure. However, this is very hard for the average person to assess.


The University of Melbourne and KPMG study found that 70% of people believe AI regulation is required, while fewer than half believe existing laws in their country are sufficient. Even if we believe AI works, we still want to know whether anybody is properly in charge. Trust in AI is really trust in the system around AI. A disclaimer saying “AI may make mistakes” is not a trust architecture.


Translating a trust architecture into practice does not require perfection but does need discipline. Here are five starting points for any organisation deploying AI.


1. Classify before you deploy

Map each AI use case against the four-tier taxonomy (low-stakes, delegated, high-stakes, intimate) before setting the level of oversight, testing and sign-off it needs. Like any business requirement, there are different levels of oversight to AI and a trust framework helps you make the case for the right one.


2. Build contestability in from day one

If a person cannot practically challenge or override an AI-influenced outcome, the system is not ready for anything beyond low-stakes use. There needs to be a system in place for human intervention if necessary.


3. Benchmark against the process being replaced

The relevant comparison is the existing process’s own error rate, delay, cost and inconsistency. The current process may not have these available. A starting point would be to benchmark or estimate the current process for an evidence-based decision.


4. Log for learning, not just compliance

Log failures in an accessible way that feeds back into the system, whether it be improved deployment or processes. Mistakes should iteratively improve future deployments, not be repeated at scale where the cost to fix will be exponentially greater.


5. Treat disclaimers as a starting point, not a policy

“AI may make mistakes” removes legal risk, but shouldn't replace a thorough testing phase, just like a deployment of any new technology or feature.


Perfect is the enemy of good


AI, like the humans it’s been trained on, may never be perfect, so what is a more honest and practical standard?


We need an evidence base to consider if the system has human oversight, reduces harm, improves judgement and makes mistakes that can be detected, contested, made accountable and learned from.


Visibility of error should not be the indicator of failure but rather the beginning of taking responsibility. The complex world we live in means most decisions, whether made by humans or AI, come with uncertainty and may turn out to be wrong. No matter how uncomfortable this may be, we need to confront this reality.


As we all wrestle with our own ideological concerns with trusting the machine, are we prepared to admit how many mistakes we were already accepting when the machine was not there to count them? We reserve the right to withhold our trust until convinced otherwise, but do acknowledge we can still make the world a better place without chasing an impossible ideal.


In Part 3 we dive a little deeper into a fundamental practical matter of trust in AI and what you can do about it i.e. transparency around what is really happening to our data.


Selected bibliography



Authors


Tai Rotem is a consulting partner at MYMAVINS with several decades experience in consumer, financial services, public health, and social research.


Reach out to him at Tai@mymavins.com.au

 







Anthony Zhang is a Senior Data Insights Consultant at MYMAVINS. An experienced economic and social researcher, Anthony expertise lays in quantitative research and data science.

 
 
 

Comments

bottom of page