Research data, by its very nature, often contains sensitive information that requires careful protection. Whether dealing with personal health records, proprietary research findings, confidential survey responses, or commercially sensitive datasets, researchers must navigate the tension between leveraging the analytical power of GenAI systems and maintaining appropriate levels of data protection. The stakes are particularly high in fields such as medical research, social sciences, and commercial research and development, where data breaches can have far-reaching consequences for individual privacy, institutional reputation, and competitive advantage. The protection of personal information within GenAI systems presents additional layers of complexity, as these systems may inadvertently memorise and subsequently reproduce sensitive personal details from their training data (Carlini et al. 2021). The phenomenon of data memorisation in large language models has been extensively documented, with researchers demonstrating that models can reproduce verbatim text from their training datasets, potentially exposing personal information, confidential communications, and proprietary content. The implications of such capabilities extend beyond immediate privacy concerns to encompass broader questions of consent, data ownership, and the ethical use of personal information in AI development.
The architecture and operational characteristics of generative AI systems create multiple pathways through which research data and personal information may be exposed or compromised. Understanding these risks requires examination of the various stages of AI system development and deployment, from initial data collection and model training through to inference and output generation. Each stage presents distinct vulnerabilities that researchers must consider when evaluating the appropriateness of GenAI tools for their specific use cases. The training phase of generative AI models represents perhaps the most significant area of risk for data exposure. During this phase, models are exposed to vast datasets that may contain sensitive research data or personal information, either intentionally included for training purposes or inadvertently captured through web scraping and data aggregation processes. The scale of modern AI training datasets, which can encompass billions of documents and data points, makes it virtually impossible to manually review all content for sensitive information. Consequently, proprietary research findings, personal communications, confidential documents, and other sensitive materials may become embedded within the model's learned representations.
The phenomenon of membership inference attacks represents a particularly concerning vulnerability in GenAI systems. These attacks enable malicious actors to determine whether specific data points were included in a model's training dataset, potentially revealing sensitive information about individuals or research subjects. Liu et al. (2024) provide a comprehensive analysis of membership inference techniques, demonstrating how attackers can exploit model outputs to infer the presence of specific data in training sets. The implications for research data protection are significant, as successful membership inference attacks could reveal participation in sensitive studies, exposure to particular treatments, or inclusion in confidential datasets. Model inversion attacks present another significant threat to data protection in GenAI systems. These sophisticated attacks attempt to reconstruct training data from model parameters or outputs, potentially enabling the recovery of sensitive information that was used during the training process (Shokri et al. 2017). The success of such attacks varies depending on the model architecture, training methodology, and the nature of the target data, but the potential for sensitive information recovery remains a persistent concern for researchers utilising GenAI technologies.
The deployment and inference phases of GenAI systems introduce additional risks related to data sharing and exposure. When researchers input sensitive data into AI systems for analysis or content generation, they may inadvertently share this information with third-party service providers, potentially exposing confidential research data to unauthorised access or misuse (Brazilian Data Protection Authority 2024). The Irish Data Protection Commission's guidance emphasises that organisations using AI products supplied by third parties face additional security and data protection risks, particularly when personal data is input by staff members into AI tools without full understanding of how such data is protected or processed (Data Protection Commission Ireland 2024). The terms of service and data handling practices of commercial AI platforms vary significantly, and researchers may not fully understand how their input data is processed, stored, or potentially used for further model training.
The Hamburg Data Protection Authority's analysis provides crucial insights into the technical architecture of AI systems, distinguishing between the storage of large language models and the processing activities that occur when personal data is input into AI-supported systems (Hamburg Commissioner for Data Protection and Freedom of Information 2024). The Authority clarifies that whilst the mere storage of an LLM does not constitute processing within the meaning of Article 4(2) GDPR, as no personal data is stored in LLMs themselves, the processing of personal data within LLM-supported AI systems must comply with GDPR requirements, particularly regarding the output of such systems. This distinction is critical for understanding where data protection obligations arise in complex AI architectures. Data leakage through model outputs represents a more immediate and observable risk in GenAI systems. These systems may inadvertently reproduce sensitive information from their training data in response to user queries, effectively creating a pathway for unauthorised access to confidential information (Ye et al. 2024). The Chinese perspective on GenAI governance, as examined by Ye et al. (2024), highlights the particular challenges posed by data leakage in systems trained on diverse, multilingual datasets that may contain varying levels of sensitive content across different jurisdictions and cultural contexts.
The global nature of many GenAI platforms introduces additional complexities related to data sovereignty and cross-border data transfers. Research data that is processed by AI systems hosted in different jurisdictions may be subject to varying legal frameworks and protection standards, potentially creating gaps in data protection coverage. The implications are particularly significant for researchers working with data subject to specific regulatory requirements or institutional policies regarding data localisation and cross-border transfers.
The development of effective risk management strategies for data protection in GenAI systems requires comprehensive approaches addressing technical, organisational, and procedural aspects whilst maintaining AI utility for legitimate research purposes. The complexity and evolving nature of GenAI technologies necessitate adaptive frameworks that respond to emerging threats. Technical safeguards form the foundation of effective data protection in GenAI systems. Privacy-enhancing technologies such as differential privacy, federated learning, and homomorphic encryption offer promising approaches for protecting sensitive information whilst enabling AI model training and deployment (Zewe 2025). These technologies enable researchers to leverage GenAI analytical capabilities whilst minimising sensitive information exposure risks. Data minimisation strategies play a crucial role by limiting personal information collection and processing to research necessities, requiring careful balance between analytical needs and privacy protection requirements. Anonymisation and pseudonymisation techniques provide additional protection layers, though modern AI systems' sophisticated capabilities challenge traditional approaches through pattern recognition and data correlation techniques (Drenik 2025).
Organisational measures represent critical components of comprehensive data protection strategies. The establishment of clear governance frameworks, data handling protocols, and accountability mechanisms ensures privacy considerations are integrated throughout the AI development and deployment lifecycle (Office of the Australian Information Commissioner 2024). These include institutional policies regarding AI use, ethics review processes, and training programmes ensuring researchers understand GenAI privacy implications. Access controls and authentication mechanisms provide essential safeguards through robust access management systems ensuring only authorised personnel access sensitive data and AI capabilities, supported by audit trails and monitoring systems. Data governance frameworks specifically designed for AI applications offer structured approaches managing privacy risks across data collection, model training, deployment, and maintenance phases. Transparency and explainability measures contribute to effective privacy protection by enabling researchers to understand how GenAI systems process and utilise their data. The development of interpretability tools and documentation standards helps researchers make informed decisions about AI application appropriateness whilst facilitating early risk identification. Regular auditing and assessment procedures provide essential mechanisms for monitoring privacy protection effectiveness and identifying emerging risks. The dynamic nature of AI technologies requires ongoing evaluation of privacy safeguards through technical security evaluations, organisational policy reviews, and emerging threat analysis.
References:
1. Brazilian Data Protection Authority. 2024. Technology Radar – Short Version in English, Number 01. Brasília, DF: ANPD. Available at: https://www.gov.br/anpd/.../radar-tecnologico-inteligencia-artificial-generativa-versao-em-lingua-inglesa.pdf ^ Back
2. Carlini, Nicholas, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, et al. 2021. “Extracting Training Data from Large Language Models.” In 30th USENIX Security Symposium (USENIX Security 21), 2633–2650. ^ Back
3. Data Protection Commission Ireland. 2024. AI, Large Language Models and Data Protection. Available at: https://www.dataprotection.ie/en/dpc-guidance/blogs/AI-LLMs-and-Data-Protection ^ Back
4. Drenik, Gary. 2025. Gen AI Struggles With Privacy—Data Protection Tech Offers a Solution. Available at: https://www.forbes.com/sites/garydrenik/2025/05/22/gen-ai-struggles-with-privacy-data-protection-tech-offers-a-solution/ ^ Back
5. Hamburg Commissioner for Data Protection and Freedom of Information. 2024. Discussion Paper: Large Language Models and Personal Data. Available at: https://datenschutz-hamburg.de/.../Discussion_Paper_Hamburg_DPA_KI_Models.pdf ^ Back
6. Liu, Yihao, Jinhe Huang, Yanjie Li, Dong Wang, and Bin Xiao. 2024. “Generative AI Model Privacy: A Survey.” Artificial Intelligence Review 58 (1): 33. ^ Back
7. Office of the Australian Information Commissioner. 2024. Can Personal Information Be Used to Develop or Train a Generative AI Model? Available at: https://www.oaic.gov.au/news/blog/can-personal-information-be-used-to-develop-or-train-a-generative-ai-model ^ Back
8. Shokri, Reza, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. “Membership Inference Attacks Against Machine Learning Models.” In 2017 IEEE Symposium on Security and Privacy (SP), 3–18. IEEE. ^ Back
9. Zewe, Adam. 2025. New Method Efficiently Safeguards Sensitive AI Training Data. Available at: https://news.mit.edu/2025/new-method-efficiently-safeguards-sensitive-ai-training-data-0411 ^ Back
