The Harms That Are Already Happening
In the first three months of 2025, Netsafe received 667 reports of sextortion threats – a 68% increase from the previous year.[1] The youngest victim was nine years old. Sextortion occurs when someone threatens to release an explicit image or video without consent unless the victim provides payment or further explicit material.
Part of this surge is driven by deepfake technology. Perpetrators no longer need stolen intimate photos; any public image can be manipulated into explicit content using freely available AI tools.
‘AI deepfake images are having a significant impact on sextortion because criminals can now produce photorealistic sexual content of a victim for the purpose of blackmail,’ says Sean Lyons, Netsafe’s chief online safety officer.[2]
While New Zealand has laws that can address some forms of digital harm, including the Harmful Digital Communications Act and Privacy Act, they were not designed for AI-generated content, struggling with its unique challenges. These include but are not limited to: consent to synthetic content, personality rights, and the automated scale of abuse.
In UNICEF’s 2025 report card on child wellbeing, New Zealand ranked 4th lowest out of 36 OECD countries, with the lowest score for mental wellbeing and the highest youth suicide rate among developed nations.[3]
While nations worldwide grapple with internet safety – regulating adult sites and social media – the AI tools driving new forms of abuse remain almost entirely unregulated, accessible to anyone, with virtually no oversight.
Why current approaches won’t work
Globally, we are seeing a push to better regulate how young people interact with the internet amidst growing concerns that unfettered access will negatively impact their social development and mental health. Nowhere has this been more exemplified than in Australia, which recently enacted a youth social media ban with the belief that restricting access to certain platforms will resolve, or at least mitigate, these concerns. Whether this approach will work remains to be seen. However, it’s important to recognise a significant blind spot in such legislation regarding AI tools. While generative AI platforms like ChatGPT, Claude, and others aren’t typically thought of as social media, they carry many of the same risks that prompted the social media restrictions in the first place. Young users can develop parasocial relationships with AI chatbots, seek validation or emotional support from systems that lack human judgment, and spend hours in isolated digital interactions that replace real-world socialisation.
Perhaps most concerning is their ability to generate deepfakes. Young people can use these tools to create non-consensual intimate images of peers, fake compromising content for bullying purposes, or fall victim to deepfakes created by others. The psychological harm of having one’s likeness manipulated, or being deceived by fabricated content of trusted figures, can be devastating for developing minds.
If the goal of these regulations is genuinely to protect young people’s development and mental health, policymakers must expand their definition of potentially harmful platforms to include AI tools that facilitate many of the same concerning patterns of use, and introduce entirely new categories of harm.
As it stands, one of the main pieces of legislation that could combat the harms of AI-generated content is the Harmful Digital Communications Act, but it has gaps that don’t adequately account for AI-based harms.
The purpose of the Harmful Digital Communications Act is to protect people from digital abuse, but it wasn’t written with AI in mind. As an example, the act provides specific protections against the non-consensual sharing of “intimate visual recordings,” defined as something “made” – implying a reality captured through a camera of a real person in a real situation. In contrast, AI-generated deepfakes are not “made” in the strictest sense; they are synthesised from other images. Therefore, a determined defendant could argue: ”I didn’t record anything. I created a synthetic image.”
Similarly, the Act requires identifying who ‘posted’ harmful content. But AI tools complicate this: most are hosted overseas, don’t verify identity, and may involve multiple actors (the prompter, the platform, the distributor). Who’s liable when an offshore AI service generates harmful content based on a NZ user’s prompt?
In place of government regulation, AI companies themselves lack any serious form of regulations or restrictions in place to protect youth. While platforms like ChatGPT and Google Gemini have a minimum user age of 13[4,5] included in their terms and conditions, no meaningful age verification mechanism is in place. Of further concern, many AI platforms
are accessible and usable without an account, which further compounds the concern about untraceable anonymous use.
What NZ Specifically Needs
New Zealand cannot afford to wait for more victims before acting. We need immediate reforms across several critical areas.
First, the criminalisation of non-consensual AI-generated intimate images. As of February 2026, there is promising progress: the Deepfake Digital Harm and Exploitation Bill is awaiting its first reading. This bill would amend both the Crimes Act 1961 and the Harmful Digital Communications Act 2015 to explicitly include AI-generated, synthesised, or altered images within the definition of “intimate visual recordings”[6]. It is important that the legislation makes it explicitly clear that this applies when minors are victims, with serious penalties for perpetrators.
If successful, this bill would finally give victims of AI-generated abuse clear legal recourse and establish serious consequences for perpetrators.
Secondly, there should be legislation that requires age verification for AI tools capable of generating images/intimate content. Oxford research has identified the online gambling industry as having valuable ‘good practice’ lessons to share regarding effective age verification implementation[7].
Gambling platforms use multi-layered approaches like age estimation from selfies, followed by document verification for borderline cases. Major gaming platforms also employ credit card verification, government ID checks, or restrict features for unverified users until parental consent is obtained. While it’s true that bypassing age controls is not difficult for determined individuals, such measures can still prevent opportunistic or casual access by minors who lack the technical knowledge or motivation to circumvent them.
Thirdly, schools need clear pathways to access support when addressing AI-generated harms. Netsafe already provides modules on the use of generative AI in class for teachers through their Education platform, The Kete. However, many schools remain unaware of these resources or how to effectively integrate them.
The Ministry of Education should actively promote Netsafe’s existing services and ensure all schools are equipped to direct students to appropriate support. Netsafe operates a free helpline (0508 638 723) that handles confidential reports of image-based abuse and deepfakes, and can assist with content removal. Additionally, schools need clear protocols for when to guide students to Netsafe versus when situations require police involvement.
Beyond these legislative and education policy changes, New Zealand needs coordinated infrastructure to handle AI harms when they occur. A dedicated team spanning police, mental health services, and education, with clear protocols for responding to AI-enabled abuse against youth.
What You Can Do Now
Whether you’re a parent, educator, policymaker, or concerned citizen, there are concrete actions you can take today to protect young people from AI-generated harms.
For Parents and Caregivers
Start having conversations about AI tools, not just social media. Many parents monitor Instagram and TikTok but don’t realise their children can access ChatGPT, Character.AI, or image-generation tools without any account or verification.
Ask your children: What AI tools are they using? Do their friends use them? Have they seen or heard about deepfakes? Approach these conversations with curiosity, not accusation. The goal is to create an environment where they’ll come to you if something concerning happens.
For Educators
You’re on the frontline of this issue. If a student reports AI-generated abuse or you suspect it’s occurring:
- Contact Netsafe: 0508 638 723 or help@netsafe.org.nz for guidance on handling image-based abuse
- Report to police at 105 for criminal matters (sextortion, threats, distribution of intimate images)
- Follow your school’s child protection protocols
For Policymakers
If you’re an MP or policy advisor, you have direct power to act. The Deepfake Digital Harm and Exploitation Bill awaits its first reading. Give it your support.
You can also take these specific actions:
- Require age verification for AI image generators – Work with the Ministry of Business, Innovation and Employment to develop regulations requiring meaningful age checks on platforms capable of generating intimate imagery.
- Fund AI incident response capacity – Allocate budget for a coordinated team across police, mental health services, and education to handle AI-related harms against youth.
For concerned citizens
Contact your local MP and the following ministers to urge them to take action:
Email template:
Subject: Urgent Action Needed on AI-Generated Harms
Dear [Minister/MP Name],
I am writing to ask you to prioritise AI safety regulations specifically:
1. Supporting the Deepfake Digital Harm and Exploitation Bill, currently awaiting first reading
2. Implementing age verification requirements for AI tools capable of generating images
3. Establishing coordinated response capability for AI-related harms to youth
Recent Netsafe data shows a 68% increase in sextortion reports, with AI deepfakes making this abuse easier than ever.
New Zealand already has the worst youth mental health outcomes in the OECD. We cannot afford to let AI tools worsen this crisis.
What specific actions will you take to address AI-generated harms against New Zealand’s Youth?
Sincerely,
[Your name]
Some ministers you may wish to contact include:
- Minister of Justice (responsible for criminal law reform)
- Minister for Digital Economy and Communications (tech policy)
- Minister of Education (school support and protocols)
For Everyone
- Share this article with parents, teachers, and policymakers in your network.
- Stay informed about the Deepfake Digital Harm and Exploitation Bill’s progress through Parliament. Track it at legislation.govt.nz and contact your MP when it comes up for a vote.
- If you know someone affected, remind them: it’s not their fault, help is available, and they’re not alone. Direct them to Netsafe (0508 638 723) for confidential support.
- Support organisations doing this work. Netsafe relies on funding to provide its services. If you’re able, consider donating or volunteering.
Conclusion
New Zealand’s young people deserve better than a regulatory vacuum. They deserve laws that account for how technology has changed, systems that can respond when harms occur, and adults willing to have uncomfortable conversations about new risks.
The tools to harm them are already here. The question is whether we’ll build the protections they need before more victims emerge.
References
[1]“Sextortion Survey Insights Report,” 2025. [Online]. Available: https://resource.netsafe.org.nz/Netsafe-Sextortion-Report-October-2025.pdf
[2]Newstalk ZB and E. de Jong, “‘Frightening’ surge in NZ sextortion with young people targeted,” Newstalkzb.co.nz, May 05, 2025. https://www.newstalkzb.co.nz/news/national/netsafe-reports-68-rise-in-sextortion-in-nz-as-ai-deepfake-threats-increase/
[3]“UNICEF Aotearoa,” Unicef.org.nz, 2025. https://www.unicef.org.nz/media-releases/new-global-data-new-zealand-ranks-alarmingly-low-for-child-wellbeing
[4]“Age requirements on Google Accounts – Google Account Help,” Google.com, 2019. https://support.google.com/accounts/answer/1350409
[5] OpenAI, “Terms of use,” Openai.com, 2023. https://openai.com/policies/row-terms-of-use/
[6]“Deepfake Digital Harm and Exploitation Bill 213-1 (2025), Members Bill Contents – New Zealand Legislation,” Govt.nz, 2025. https://legislation.govt.nz/bill/member/2025/0213/latest/LMS1536414.html (accessed Jan. 06, 2026).
[7]“OII | Effective Age Verification Techniques: Lessons to be Learnt from the Online Gambling Industry,” Ox.ac.uk, 2017. https://www.oii.ox.ac.uk/research/projects/effective-age-verification-techniques/