In August 2023, a meal bot deployed by a major New Zealand supermarket chain suggested dangerous, toxic, and bizarre recipes to customers.
In July 2024, a 72-year-old grandmother from Taranaki lost $224,000. She’d been shown a video of Prime Minister Christopher Luxon promoting a crypto investment.
In September 2024, New Zealand universities deployed AI detection tools, which flagged thousands of student assignments as AI-generated. However, following widespread backlash over high false-positive rates and inherent unreliability, several institutions have backtracked and discontinued the software.
In May 2025, MP Laura McClure highlighted how easy it was to create a compromising deepfake of a person holding up an AI-generated nude image of herself in parliament.
In February 2026, the New Zealand Supreme Court formally warned that submitting hallucinated AI citations could constitute contempt of court or obstruction of justice.
In June 2026, NCSC alongside other 5 Eyes organisations, put out an advisory stating “AI is not a future consideration; it is already here. It lowers barriers for malicious actors and increases the speed and complexity of attacks.”
“AI is not a future consideration; it is already here. It lowers barriers for malicious actors and increases the speed and complexity of attacks.”
It might seem like AI development is something happening elsewhere, in labs we’ll never see, built by companies with no presence here. But we live in an integrated world. New Zealand doesn’t need to have a frontier AI lab for AI and its risks to have an impact here. AI is being picked up by Kiwis across the country to answer questions and provide advice, by businesses to streamline work, by our institutions. It’s also being used to scam Kiwis of their life savings, create intimate deepfakes of people to extort, it’s being used to make lasting decisions on someone’s employment, loan, insurance and increasingly it’s also being used to cut corners in education.
With such widespread and rapid adoption, we’re concerned that not enough people appreciate the risks AI poses, or how lasting its impact on Aotearoa could be. We’ve seen this story before: the world is only now, twenty years on, reckoning with what social media did to us with countries scrambling to legislate youth access to it. We don’t want AI to be the same, understood only in hindsight. This page is our attempt to get ahead of it.
Here we cover six key areas of concern:
Personal Harm & Scams
Compared with other countries, New Zealand has always been more vulnerable to scams. A 2018 study covering 16 countries found Kiwis reported the highest exposure to tech support scams of anyone surveyed, and interestingly, the group most impacted wasn’t the elderly. It was young, male internet users who were more confident online and therefore less cautious.
The common advice for avoiding scams has always been built around looking for obvious tells: was the email actually from your bank’s address? Was the grammar off? Did that text or DM from a friend actually sound like them? If you called them back, would you recognise their voice?
Additionally, pulling off a convincing scam took effort. A scammer had to tailor a phishing message by hand or send out more generic messages in bulk, trawl the internet for vulnerabilities, and targeting someone specifically took slow, painstaking work; most attacks were opportunistic rather than aimed at one person.
Now, with modern AI, scammers have been handed a bazooka. With a few posts scraped from social media, the voice of the CEO of a bank can be faked, or your doctor, or even your relative. Deepfake video calls can put a real face and a real voice on the other end of a call in real time, and phishing emails that once took a scammer an evening to write can now be generated in seconds, personalised with your name, your employer, even your recent purchases. What used to require patience, technical skill, and a decent chunk of manual labour can now be done by anyone with a laptop and an afternoon.
As AI models become more and more accessible and capable, the advice on how to identify and avoid a scam doesn’t hold up anymore, because AI has specifically solved for every tell we’ve been taught to look for. The grammar is flawless now; the domain can be spoofed, and the voice on the other side of the phone does sound like someone you trust. In New Zealand, scammers have already used an AI-generated deepfake of Westpac NZ’s chief executive in investment ads, and cloned the face of a real Auckland eye specialist to sell a fake health product.
If this technology can be weaponised against individuals so effortlessly, it can be scaled to target institutions. It is worth remembering that 2026 is an election year in New Zealand. We’ve already seen crude, obviously fake AI content in our politics: cartoon-style attack ads and deepfake videos of politicians circulating on bogus Facebook news pages. What happens when someone with genuine skill and motivation deploys a version that isn’t obviously fake?
In short: the tells we were taught to watch for don’t work anymore, and 2026 puts that to the test at national scale.
Trust & Truth
Spend a few minutes on any comment section, on any platform, and you’ll see the same statement over and over: “that’s just AI,” “fake,” “not real.” Scroll for long enough, and it’s not hard to understand why. Social media is increasingly flooded with AI-generated content, some of it made by bots, some by people chasing engagement, and all of it adding to a growing pile of slop.
This same distrust in content isn’t just isolated to the internet; we’re seeing the same in our creative spaces. Authors and artists are having their publications put into question. Just last year, one of Aotearoa’s largest book competitions disqualified two authors after their book covers were found to be AI-generated, bringing the whole process into question.
We’re also seeing this in our classrooms and campuses across Aotearoa. Universities and schools are concerned about whether students are actually completing assignments or if they’re just getting a chatbot to do it for them. This has led to schools using AI detection tools like Turnitin to catch cheaters, but these tools aren’t always accurate and can result in false positives. In response, some students have started recording themselves completing assignments, just to have proof ready if they’re wrongly accused. We’ve written more about what this means for students and educators in our guide on AI and assignments.
Altogether, what this all means is we’re entering a space where the very existence of believable AI content is bringing into question people’s abilities, and work. This is called the liar’s dividend. It’s the idea that once convincing fakes exist, anyone can dismiss something true as a lie. This works in both directions. Fake things can get believed. True things get disbelieved. As AI tools that generate images, video, and text keep improving, the small tells that once gave a fake away are disappearing. It’s only going to get harder to tell the difference, not easier.
New Zealand is a small country; we’re famously two degrees of separation from anyone else here. Something true or false rarely stays contained to a screen; it moves through communities of people who know each other, work together, live down the road from one another. Being able to agree on basic, shared facts isn’t a nice-to-have in a country like ours. It’s part of how we function. So what happens to that when nobody can agree on what’s real anymore?
In short: in a country this small and this connected, losing a shared sense of what’s real isn’t an abstract harm; it’s a functional one.
Being Judged by Systems
Because of how general purpose modern AI LLMs are, it’s very easy and tempting for organisations to slot them into places where you might normally have a person doing some kind of repetitive task. It just so happens that these tasks can sometimes involve things which are very personal. Why have your HR team spend hours reviewing 1,000 job applications to filter out the junk or unqualified ones, when AI can do it for you in a few minutes?
Why have a teller manually review every loan application and comb through someone’s banking history, when you can feed their information into a model that gives them a score instantly?
The thing is, as much as this can save time, AI systems making decisions about people based on information from those people can fail because of two things: bias and visibility.
Bias: The data used to train these systems has an enormous impact on the decisions they make. An AI trained to recognise “good” spending habits in the United States may misread or incorrectly classify people here. Depending on where the data was sourced, it can also carry the internalised biases of that source. Train a model to identify what a good applicant looks like by feeding it the profiles of your historic employees, and you can end up accidentally training it to be sexist if your workforce is mostly male, or to only favour people from certain postcodes. As strange as that sounds, it isn’t the most obvious failure mode, and it’s worth remembering that the training data these systems use tends to come from the United States, which can have a profound effect when applied in an NZ context.
Visibility: Modern AI systems are trained on datasets so large, and run on neural networks so complex, that in some cases no single person can say for certain how they arrive at a decision, not even the people who built them. So when an AI makes a decision about a person, it’s not always clear how it got there.
As one Victoria University AI researcher put it, this isn’t a hypothetical risk, it’s already happening. Project Employ, which helps neurodiverse New Zealanders find work, is a case in point. They say AI CV screening specifically disadvantages their candidates: people who might describe their own skills differently, or not use the exact keywords a system is scanning for, even when they’re perfectly capable of doing the job. It’s a clear example of the bias problem above, a system trained on how one “kind” of applicant presents themselves ends up penalising anyone who presents differently.
In May 2026, Parliament passed the Social Security (Modernisation) Amendment Bill under urgency, letting the Ministry of Social Development use automated systems to help make benefit decisions. MSD has said the systems involved would be rules-based rather than generative AI like ChatGPT. The government says safeguards will cover it: human oversight, and requirements to manage bias and keep the process transparent. Critics point to two cautionary tales. Amsterdam spent five years and hundreds of thousands of euros trying to build a demonstrably fair welfare fraud system, with extensive bias testing and community consultation, and it still produced discriminatory outcomes. Then there’s Australia, where the Robodebt scheme automatically demanded welfare repayments based on flawed calculations. Surveys suggest around two-thirds of New Zealanders already have real concerns about AI being used this way.
This isn’t limited to hiring and welfare. Banks and insurers are exploring AI-driven scoring for loans and cover. Housing decisions are next in line. Wherever a decision affects someone’s income, their home, or their access to support, the same two problems follow: the process is opaque, and the bias, when it exists, is often invisible even to the people running the system.
New Zealand’s answer to this so far has been the Algorithm Charter, a voluntary commitment from government agencies to manage these risks carefully. A charter is a good start. But it is not the same as an outcome. The question isn’t whether these systems can be built responsibly. It’s whether they’re being checked closely enough, on the people they’re already being used on, right now.
In short: the safeguards exist on paper; the real question is whether anyone is checking that they hold up on the people already living under them.
The Human Impact
More people are turning to AI chatbots for company, not just help with tasks. For someone isolated, or without easy access to people to talk to, that can feel like a real lifeline, and sometimes it genuinely is. AI has real potential in this space. But that potential has outpaced the conversation about its risks.
Part of how today’s AI chatbots are trained is called reinforcement learning from human feedback: a human reviewer scores different responses, and the model learns to produce whatever scores well. It just so happens that human reviewers tend to score agreeable answers higher, which trains the AI to be, well, agreeable.
This means chatbots can mirror what you tell them and tend to validate it rather than push back, a trait called sycophancy.
For most people, it’s mildly annoying at worst when an AI tells you an obviously bad idea is actually the best thing it’s ever heard.
But for some people, especially those already vulnerable to distorted thinking, like the isolated, sleep-deprived, or predisposed to psychosis or mania, an endlessly available conversational partner that says “that’s a great idea” can reinforce and accelerate a spiral. Clinicians have started calling this AI psychosis. It isn’t a formal diagnosis yet, but it’s a real and growing concern, serious enough that multiple families have taken chatbot companies to court over it, alleging their systems worsened a mental health crisis.
This matters even more in Aotearoa than it might elsewhere, because our mental health system is already stretched thin. Nearly a third of general practices had closed their books to new patients by early 2025. Cost is now the single biggest reason New Zealanders go without counselling, and access for young people specifically has kept getting worse even as other wait times slowly improve.
AI isn’t the first technology to have a profound impact on people’s mental health. Countries are only now moving to restrict how young people use social media. Australia banned it for under-16s in December 2025; the UK and France have both since announced plans to follow. A Los Angeles jury recently found that Meta and YouTube’s addictive design features involved malice, oppression, or fraud.
Social media took a generation to be reckoned with, and by the time it was, an entire cohort had already grown up inside the experiment. AI doesn’t offer us that same run-up: it’s moving faster, embedding itself more deeply into daily routines, and showing up at moments, grief, isolation, crisis, that social media rarely touched as directly. Waiting to see how this plays out isn’t really a neutral choice.
If a crisis like Covid-19 happened again, with the same extended isolation, disrupted routines, and widespread anxiety, but this time with today’s AI companions available, what would we see emerge? We don’t know.
This is why we need to have the conversation now, before we’re forced to have it in hindsight.
In short: AI companionship is already reshaping mental health in ways we don’t fully understand yet, in a country whose support system is already stretched thin.
If you or someone you know is struggling, Need to Talk (call or text 1737) is free and available any time.
Economic Disruption
In June 2026, a New Zealand model accused a major streetwear brand of using AI to recreate his likeness in a marketing campaign without paying him or asking permission (an allegation the brand denies). An independent investigation the brand commissioned found no evidence his facial features had been used, though the model has rejected that finding.
The impact of AI on work is really hard to measure because its use is still relatively new across the workforce. That being said more and more stories like this are popping up where creatives are having their work displaced by AI.
An AI-generated song topped the NZ iTunes charts, and APRA AMCOS, the body that represents New Zealand songwriters and composers, confirmed the AI platforms behind tracks like it were trained on scraped songs without the original artists’ permission. Musician Bic Runga makes the point: she doesn’t want to see New Zealand outsource its creative disciplines to a machine, singling out waiata, te reo Māori, and haka as things she doesn’t want fed into a system and spat back out. A country our size doesn’t have an endless supply of another Bic Runga, or another generation of songwriters carrying that same knowledge forward. When something this specific and this local gets can be automated away, it isn’t easily replaced.
A country our size doesn’t have an endless supply of another Bic Runga, or another generation of songwriters carrying that same knowledge forward.
The literary world has already started drawing its own line. The Ockham Book Awards disqualified two authors in late 2025, ahead of the 2026 awards, after their publisher used AI-generated cover art, a clear signal that at least some New Zealand institutions think consent and authorship still matter, even when the tool makes something faster or cheaper.
What connects all three of these isn’t really about AI being capable of making music, art, or images. It’s that in every case, someone’s work, someone’s face, or someone’s voice was used to replace or undercut them.
This isn’t limited to creative work. Estimates suggest as many as one in 10 New Zealand jobs could face near-term automation risk. Whatever the exact number turns out to be, it’s moving faster than most workplaces, or training systems have had time to adjust to.
The question isn’t whether AI will change the New Zealand job market. The question is whether the people affected get a say in how, or whether it just happens to them, the same way it happened to a model, a chart full of musicians, and two authors who did nothing wrong except publish a book with the wrong publisher.
In short: whether it’s a model’s face, a songwriter’s melody, or an author’s cover art, AI is already displacing New Zealanders’ work; the open question is whether they get any say in it.
Reliability at Scale
Every system fails. No process or program works as intended 100 percent of the time, not the human ones, and not the automated ones. But what happens when you deploy something that isn’t specialised for just the task it’s completing, where you can’t reliably say what could go wrong? This is one of the pitfalls of LLMs: they’re trained on basically the entire internet, and capable of producing outputs in all sorts of domains. Organisations deploy them inside a shell of restrictions, meant to answer questions in just one area, or only give a certain type of response. But unlike more specialised tools, which can literally only produce specific outputs, an LLM can do so much more, and what it takes to break its restrictions and guardrails isn’t always obvious.
New Zealand has already seen what that looks like. PAK’nSAVE’s AI meal-planning bot suggested a recipe that would produce chlorine gas, describing it as refreshing. It wasn’t jailbroken with some clever technical exploit: someone simply typed in bleach, ammonia, and water as their available ingredients, and the bot complied.
Then there’s the other kind of failure. Security testers recently got Heidi, an AI scribe used across New Zealand emergency departments, to bypass its own safety instructions and generate information it should never produce, simply by rewriting how they talked to it. This one was caught. It happened during controlled testing, and it never reached a real patient.
Put those two side by side and the question that emerges is this: should we be putting a system that knows the recipe for chlorine gas in front of people looking for a meal? Should we be giving a system patient data when all it takes is giving it just the right phrase to trick it into giving you that data? This has already happened on systems Kiwis rely on, and the only reason no harm has been done is because we were lucky and someone noticed.
As more New Zealand institutions, organisations, and businesses adopt AI systems, more and more Kiwis are exposed to the failures these systems can have. The only way we have a chance of avoiding this is if our institutions figure out ways to use these tools that don’t put people in the blast radius when they fail: independent testing before deployment, real accountability when something goes wrong, and enough transparency that the next chlorine gas recipe or jailbroken scribe gets caught before it reaches anyone at all.
In short: these systems will keep failing. The only real question is whether anyone catches it before it reaches a real person.
Where This Leaves Us
None of these concerns and risks are hypothetical, and none of them require New Zealand to have an AI lab. A grandmother’s savings, a job application nobody explained, a chatbot that agreed with the wrong thing at the wrong moment, a system that failed and one that was only caught by luck and good testing. Different shapes, same underlying problem: these tools are already woven into daily life here, and the systems meant to catch it when they go wrong are still being built, tested, and in some cases, not built at all.
Something needs to change, and the most direct place to start is asking the institutions already using these tools to be accountable for them. If your bank, your insurer, or your employer is using AI to make decisions about you, you’re entitled to ask what oversight exists. If your local MP hasn’t turned their mind to what AI governance in New Zealand should look like, they should hear from someone who has. We’ve built tools to make both of those things take two minutes, not an afternoon.
There’s also a lot you can do closer to home. Agree on a family code word so a deepfake scam won’t work on you. Ask an AI system for its source before you act on what it tells you. These are small habits, but the kind that hold up regardless of how good the technology gets. See our full list of ways to take action.
New Zealand didn’t build the frontier AI labs shaping this technology, and we’re not going to. But we’ve been here before. We didn’t invent nuclear weapons either, but that didn’t stop us from taking a clear, early, principled stand on them when it mattered, well before it was the easy or obvious thing to do. We can do the same here. Not by building the biggest AI systems, but by building the most informed public, the most carefully governed institutions, and the clearest example of what it looks like to bring a powerful technology into a country’s daily life without sleepwalking into harm. That’s a kind of leadership our country has a chance to deliver. It just requires deciding to.