Conducting AI System Privacy Assessment

Overview

AI systems that process personal data require a combined privacy and conformity assessment addressing both GDPR obligations and the EU AI Act (Regulation 2024/1689). This skill integrates the GDPR Art. 35 DPIA framework with AI-specific risk assessment, encompassing training data lawfulness, Art. 22 automated decision-making implications, algorithmic fairness, and the NIST AI Risk Management Framework MAP function. The assessment methodology draws from the EDPB-EDPS Joint Opinion 5/2021 on the AI Act proposal and subsequent EDPB Guidelines 06/2025 on AI and data protection.

Legal Framework

GDPR Provisions Applicable to AI

| Provision | Application to AI Systems | |-----------|--------------------------| | Art. 5(1)(a) — Lawfulness, fairness, transparency | AI processing must have a lawful basis; the logic of AI decisions must be explainable to data subjects | | Art. 5(1)(b) — Purpose limitation | Training data collected for one purpose cannot be used to train AI models for an incompatible purpose without further lawful basis | | Art. 5(1)(c) — Data minimisation | AI models should not require more personal data than necessary; synthetic data and anonymisation should be considered | | Art. 5(1)(d) — Accuracy | AI outputs affecting individuals must be accurate; model drift must be monitored | | Art. 6(1) — Lawful basis | Each stage of AI processing (data collection, model training, inference, output use) requires a lawful basis | | Art. 9 — Special categories | Training on health, biometric, genetic, racial, political, religious, sexual orientation, or trade union data requires an Art. 9(2) exemption | | Art. 13-14 — Transparency | Privacy notices must disclose the existence of automated decision-making, meaningful information about the logic involved, and the significance and envisaged consequences | | Art. 22 — Automated decision-making | Data subjects have the right not to be subject to decisions based solely on automated processing that produce legal effects or similarly significantly affect them, with exceptions under Art. 22(2) | | Art. 25 — Data protection by design | AI systems must embed privacy protections from the design phase: privacy-preserving ML techniques, differential privacy, federated learning | | Art. 35 — DPIA | AI systems meeting EDPB WP248rev.01 criteria (evaluation/scoring, automated decision-making, innovative technology) require a DPIA |

EU AI Act (Regulation 2024/1689) Risk Classification

| Risk Category | Description | AI Act Requirements | GDPR Overlap | |--------------|-------------|---------------------|--------------| | Unacceptable Risk (Art. 5) | AI practices prohibited outright: social scoring by public authorities, real-time remote biometric identification in public spaces (with exceptions), emotion recognition in workplace/education, untargeted scraping for facial recognition databases | Prohibited — cannot be deployed | Art. 9 special categories, Art. 35 DPIA | | High Risk (Annex III) | AI systems in listed areas: biometrics, critical infrastructure, education, employment, essential services, law enforcement, migration, administration of justice | Conformity assessment, risk management system, data governance, transparency, human oversight, accuracy/robustness/cybersecurity | Art. 22, Art. 35 DPIA, Art. 25 DPbD | | Limited Risk (Art. 50) | AI systems with specific transparency obligations: chatbots, emotion recognition, deep fakes | Transparency obligations (notify users they are interacting with AI) | Art. 13-14 transparency | | Minimal Risk | All other AI systems | No specific AI Act obligations; voluntary codes of practice | Standard GDPR compliance |

EDPB-EDPS Joint Opinion 5/2021

Key recommendations from the Joint Opinion on the AI Act proposal:

1. Prohibition of remote biometric identification in public spaces — supported without exceptions
2. AI-specific DPIA — the EDPB recommended mandatory DPIAs for all high-risk AI systems, not limited to Art. 35 triggers
3. Right to explanation — Art. 22(3) right to obtain human intervention and contest decisions should apply to all AI-assisted decisions, not only fully automated ones
4. Training data governance — controllers must ensure lawfulness of personal data used for AI training throughout the model lifecycle, including when training data is subsequently deleted
5. Purpose limitation for AI training — repurposing personal data for AI model training requires a compatibility assessment under Art. 6(4) or a separate lawful basis

Assessment Methodology

Phase 1: AI System Classification (Week 1)

1.1 AI Act Risk Classification

Determine the AI system's risk category under the AI Act:

1. Check Art. 5 prohibited practices list — if the system falls within a prohibition, it cannot be deployed.
2. Check Annex III high-risk categories:

• Biometric identification and categorisation of natural persons
• Management and operation of critical infrastructure
• Education and vocational training (admissions, assessment, proctoring)
• Employment, worker management, and access to self-employment (recruitment, task allocation, performance evaluation, termination)
• Access to and enjoyment of essential private and public services (credit scoring, insurance pricing, emergency services dispatch)
• Law enforcement (crime risk assessment, polygraph, evidence evaluation)
• Migration, asylum, and border control
• Administration of justice and democratic processes

1. Check Art. 50 transparency obligations for limited-risk AI.
2. If not high-risk or limited-risk, classify as minimal risk.

1.2 GDPR Processing Assessment

1. Identify all personal data processed at each AI lifecycle stage: data collection, data preparation, model training, model validation, inference/deployment, output use.
2. For each stage, identify the lawful basis under Art. 6(1).
3. If special category data is involved, identify the Art. 9(2) condition.
4. Assess whether Art. 22 applies (solely automated decisions with legal or significant effect).

Phase 2: Training Data Lawfulness (Week 2)

2.1 Data Collection Lawfulness

| Assessment Question | Requirement | |--------------------|-------------| | Was the training data collected with a lawful basis under Art. 6(1)? | Each data source must have an identified lawful basis | | Is the use of data for AI training compatible with the original collection purpose? | Art. 6(4) compatibility assessment or Art. 5(1)(b) further processing analysis | | Was consent obtained for AI training specifically? | If relying on Art. 6(1)(a), consent must be specific, informed, and freely given for the AI training purpose | | If using legitimate interest, has a balancing test been conducted? | Art. 6(1)(f) requires documented LIA including AI-specific impacts | | Does training data include special categories? | Art. 9(2) exemption required; Art. 9(2)(j) scientific research may apply with safeguards |

2.2 Data Quality and Bias Assessment

| Assessment Area | Requirements | |-----------------|-------------| | Representativeness | Training data must be representative of the population the AI system will be applied to. Underrepresentation of demographic groups must be identified and addressed. | | Label accuracy | If supervised learning, labels must be accurate and free from historical bias. Human labellers must be trained on anti-discrimination principles. | | Temporal validity | Training data must reflect current conditions. Stale training data can produce discriminatory outputs. | | Proxy variables | Identify features that may serve as proxies for protected characteristics (postcode as proxy for ethnicity, name as proxy for gender). | | Data provenance | Document the source, collection methodology, and processing history for all training data. |

2.3 Training Data Retention

• Art. 5(1)(e) storage limitation applies to training data.
• Once training is complete, assess whether original training data must be retained or can be deleted.
• If training data is deleted, document how data subject rights (access, erasure) will be facilitated regarding data embedded in the trained model.
• Consider whether the trained model itself constitutes personal data (if individual data points can be extracted through model inversion attacks).

Phase 3: Art. 22 Automated Decision-Making Assessment (Week 3)

3.1 Art. 22(1) Applicability Test

Does the AI system make decisions about individuals?
├─ NO → Art. 22 does not apply.
└─ YES → Continue.
    │
    Are decisions based solely on automated processing?
    ├─ NO → Art. 22(1) does not apply, but Art. 13-14 transparency still applies.
    │   (Note: "meaningful human involvement" must be genuine, not rubber-stamping.)
    └─ YES → Continue.
        │
        Do decisions produce legal effects or similarly significantly affect the individual?
        ├─ NO → Art. 22(1) does not apply.
        └─ YES → Art. 22(1) applies. The individual has the right not to be subject
            to the decision unless an Art. 22(2) exception applies.

3.2 Art. 22(2) Exceptions

| Exception | Requirements | |-----------|-------------| | Art. 22(2)(a) — Necessary for contract | Decision must be necessary for entering into or performing a contract with the data subject | | Art. 22(2)(b) — Authorised by law | Union or Member State law must authorise the decision and provide suitable safeguards | | Art. 22(2)(c) — Explicit consent | Data subject has given explicit consent to the automated decision |

3.3 Art. 22(3) Safeguards

Even where an Art. 22(2) exception applies, the controller must implement suitable measures including:

• Right to obtain human intervention
• Right to express the data subject's point of view
• Right to contest the decision
• Meaningful information about the logic involved (not the full algorithm, but sufficient to understand the factors and their relative weight)

Phase 4: Algorithmic Bias and Fairness Assessment (Week 3-4)

4.1 Fairness Metrics

| Metric | Description | Application | |--------|-------------|-------------| | Demographic parity | Positive outcome rates should be equal across protected groups | Credit scoring, hiring | | Equalized odds | True positive and false positive rates should be equal across groups | Criminal risk assessment, fraud detection | | Predictive parity | Positive predictive value should be equal across groups | Medical diagnosis, recidivism prediction | | Individual fairness | Similar individuals should receive similar outcomes | Loan pricing, insurance premium calculation | | Counterfactual fairness | Outcome should not change if only the protected characteristic changes | Any decision-making system |

4.2 Bias Testing Protocol

1. Define protected characteristics relevant to the jurisdiction and use case (race, gender, age, disability, religion, sexual orientation per Equality Act 2010 / EU Charter Art. 21).
2. Partition test dataset by protected characteristics.
3. Calculate fairness metrics for each group.
4. Identify disparate impact: if the selection rate for any protected group is less than 80% of the rate for the group with the highest rate (four-fifths rule from US EEOC, widely adopted as a starting benchmark).
5. If disparate impact is detected, assess whether the disparity is justified by a legitimate, non-discriminatory factor.
6. Document bias testing methodology, results, and remediation actions.

Phase 5: NIST AI RMF MAP Function Integration (Week 4)

The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) MAP function identifies the context, capabilities, and potential impacts of AI systems. Integrate the following MAP subcategories:

| Subcategory | Assessment Action | |-------------|-------------------| | MAP 1.1 | Document the intended purpose, context of use, and deployment environment | | MAP 1.2 | Document interdependent and interconnected systems | | MAP 1.5 | Identify intended users, affected individuals, and stakeholders | | MAP 1.6 | Assess impacts on individuals, groups, communities, organisations, and society | | MAP 2.1 | Establish the AI system's knowledge limits and conditions where it may fail | | MAP 2.2 | Document scientific integrity of training and testing methodologies | | MAP 2.3 | Assess environmental impact of AI system training and deployment | | MAP 3.1 | Document potential benefits of the AI system | | MAP 3.2 | Document potential costs, risks, and negative impacts | | MAP 3.4 | Map risks specifically to affected communities and stakeholders | | MAP 3.5 | Document likelihood and severity of identified risks | | MAP 5.1 | Engage with diverse stakeholders and affected communities | | MAP 5.2 | Engage with domain experts, AI practitioners, and sociotechnical experts |

Phase 6: Combined Risk Assessment and Reporting (Week 5-6)

1. Consolidate GDPR DPIA risks, AI Act compliance gaps, and NIST AI RMF findings.
2. For each risk, assess likelihood and severity using the DPIA risk matrix.
3. Identify mitigation measures combining privacy controls and AI governance controls.
4. Calculate residual risk levels.
5. If high-risk AI under the AI Act, prepare conformity assessment documentation.
6. Submit to DPO for Art. 35(2) advice.
7. If residual risk remains high, initiate Art. 36 prior consultation.

AI-Specific Privacy Risks

| Risk Category | Description | Mitigation Approach | |---------------|-------------|---------------------| | Model inversion | Attacker reconstructs training data from model outputs | Differential privacy during training, output perturbation, access controls on model API | | Membership inference | Attacker determines whether a specific individual's data was in the training set | Regularisation, differential privacy, limiting model confidence scores | | Data poisoning | Malicious manipulation of training data to bias model outputs | Training data provenance verification, anomaly detection, robust training techniques | | Concept drift | Model accuracy degrades over time as real-world data distribution changes | Continuous monitoring, automated retraining triggers, human review of edge cases | | Explanation manipulation | Gaming of AI explanations to hide discriminatory factors | Multiple explanation methods, adversarial testing of explanation consistency | | Feedback loops | AI decisions create data that reinforces existing biases | Regular bias auditing, human-in-the-loop review, outcome monitoring by demographic group |

Common Assessment Deficiencies

1. Treating AI training as a single processing operation: Each stage (collection, preparation, training, inference) is a distinct processing operation requiring its own lawful basis assessment.
2. Relying on anonymisation claims without verification: Data claimed to be anonymised may be re-identifiable through model inversion or linkage attacks.
3. Inadequate Art. 22 human involvement: Rubber-stamp human review of AI decisions does not constitute meaningful human involvement.
4. Ignoring purpose limitation for training data: Using customer data collected for service delivery to train AI models without compatibility assessment.
5. No bias testing on protected characteristics: Fairness assessment limited to overall accuracy without demographic disaggregation.
6. Static assessment: AI systems evolve through retraining; the assessment must be updated when the model is retrained or the deployment context changes.

Enforcement Precedents

• Italian Garante vs Clearview AI (2022): EUR 20 million fine for scraping biometric data from the internet to train facial recognition AI without lawful basis, consent, or DPIA.
• Italian Garante vs Deliveroo (2021): EUR 2.5 million fine for algorithmic management of food delivery riders constituting Art. 22 automated decision-making without adequate safeguards. The algorithm assigned reputation scores and allocated delivery shifts without meaningful human review.
• Hungarian DPA vs Budapest Bank (2023): Fine for automated credit scoring system that used postcode as a proxy for ethnicity without bias impact assessment.
• CNIL vs Clearview AI (2022): EUR 20 million fine; CNIL ordered deletion of all data of French residents and prohibited further collection.
• Dutch DPA vs Tax Authority (2020): EUR 3.7 million fine for algorithmic profiling system (SyRI) that disproportionately targeted ethnic minorities in fraud detection, violating ECHR Art. 8 and Art. 14.