Why AI purchasing has to pass the privacy test first

The student names, classes, grades, assignments and even behaviour records a school handles every day all count as personal data under the Personal Data (Privacy) Ordinance. AI tools carry two particular risks. First, the data usually sits in the cloud outside the school, sometimes outside Hong Kong. Second, some services use what users type to train their models. Once a student's essay goes into training data, there is no getting it back. Digging out the contract when a parent asks is already too late.

As the AI for Empowering Learning and Teaching Funding Programme brings AI into more subjects, the range of student data involved will only widen. Set up a screening process on the first purchase and you can reuse it on every one after that.

The six data protection principles applied to AI

  • Principle 1 (purpose and manner of collection): collect only the minimum data teaching needs. If a class number will do instead of a full name, use it, and anonymise wherever you can.
  • Principle 2 (accuracy and retention): confirm with the vendor how long data is kept, when it is deleted after the contract ends, and whether the school can ask for deletion on its own initiative.
  • Principle 3 (use limitation): data may be used only for the purpose stated when it was collected. Whether it is used to train models must be answered in writing; if it is, you need separate consent, or choose an option where that function can be switched off.
  • Principle 4 (security): confirm encryption, access permissions and where the servers are; storage outside Hong Kong needs an assessment of the level of protection there.
  • Principle 5 (transparency): update your school's own privacy policy so parents know which AI tools the school uses and what data they handle.
  • Principle 6 (access and correction): confirm the school can, at a parent's or student's request, access and correct the personal data held in the tool.

Ten questions to ask the vendor

  • In which region are the servers that store student data?
  • How long is data kept? How soon after the contract ends is it deleted? Can they provide proof of deletion?
  • Is what students type used to train AI models? Can that be switched off?
  • Which third parties, such as cloud services and model providers, have access to the data?
  • Is the data encrypted in transit and at rest?
  • Can a school administrator control the access permissions of teachers and students?
  • If there is a data breach, what is the timeframe and procedure for notifying the school?
  • Can students use it without providing their real names?
  • Has the service had an independent security audit? Can they provide a summary?
  • Can the answers above be written into the contract or confirmed in writing?

Special considerations for student data

Students are minors, and the bar for protecting their personal data should be higher. Three arrangements work in practice. First, avoid having students sign up to public AI services with a personal email; use a school-managed account system or have the teacher operate the tool on their behalf. Second, when a new type of data is collected, voice or images for instance, notify parents and obtain consent through the school's established procedure. Third, set clear rules of use for students, including not typing their own or their classmates' personal data into an AI tool. That is a lesson in AI literacy in its own right.

The Office of the Privacy Commissioner for Personal Data has published a model framework and guidance on artificial intelligence. Schools can look up the latest version on the Commissioner's website as the basis for their own policy.

Turning the checklist into a purchasing process

Turn the ten questions above into a standard form, have the vendor answer them in writing before any AI tool is bought, and file it with the quotation. Anyone who cannot answer, or will not put it in writing, loses marks or is out. The form later serves three purposes: a ready answer when a parent asks, proof of responsibility if there is a breach, and evidence in the programme report that the school assessed the tool with care.