𝛑
𝛑
Posts List
  1. 0x00 Introduction
  2. 0x01 A Short History of Generative AI
  3. 0x02 Regulation and Compliance
    1. 1. From GDPR to the EU AI Act
    2. 2. Chinese AI Regulation
  4. 0x03 An LLM Security Framework
    1. 1. Data Privacy Protection
    2. 2. Model Security
    3. 3. Network Security
  5. 0x04 Case Studies
    1. 1. Infrastructure Data Leakage
    2. 2. Who’s God in Here?
  6. 0x05 Wrapping Up
  7. Appendix: References

A Practical Security Guideline for Generative AI

Feb 24, 2025 Felix Zhao 中文 4.2k words ~26 min read

This piece started life as an internal draft — a “Generative AI Security Guideline” I wrote for the company I work at. What follows is the public version, rewritten for a broader audience but keeping the same structure and the same opinions.

0x00 Introduction

Generative AI has stopped being a novelty. Between ChatGPT’s public launch and DeepSeek R1’s release in January 2025, LLMs went from “interesting demo” to “thing that reshapes valuations of Chinese tech stocks in a single week.” Every major Chinese internet company now ships its own foundation model — Qwen (Alibaba), ERNIE (Baidu), Kimi (Moonshot), Doubao (ByteDance) — and every hyperscaler has raced to bolt DeepSeek R1 support onto its inference platform: AWS, Alibaba Cloud, Tencent Cloud, Volcengine. Traditional finance shops that spent 2023 in wait-and-see mode are now actually running pilots.

That’s exciting. It is also the part where a security architect starts to sweat. Users get productivity gains; the enterprise inherits an entire new class of risks that most existing controls do not cover. The point of this document is to walk through those risks — the regulatory framing, the actual threats, the controls that work — with a bias toward what you can ship in production rather than what makes for good conference slides.

I have tried to keep the register conversational and skip the heavy math. AI moves fast enough that anything I write about specific tooling will be partly stale by the time you read it; the framework and the threat model should age better than the vendor names.

0x01 A Short History of Generative AI

Generative AI sits at the current frontier of ML, but the foundations are older than most people realize. Markov chains in the 1950s and Hidden Markov Models (HMMs) in the 1970s set up the probabilistic backbone. From there, the field crawled through Naive Bayes, k-NN, and SVMs — the “classical ML” era. Then compute got cheap, CNNs opened up deep learning, and the sequence-modeling story went from CNN to LSTM to GRU, each iteration trading off memory horizon and parameter cost. Google’s Transformer paper — the famous “Attention Is All You Need” — plus BERT are what actually got us to today’s LLM zoo. That is the technical bedrock for basically every modern foundation model.

img
(Figure from LLM Survey)

Early LLM work stayed inside academia and a few big labs. Then OpenAI released GPT (Generative Pre-trained Transformer) in 2018 and the wider world noticed. Large-scale pre-training plus autoregressive generation cracked open text generation, dialogue, content creation, customer service. GPT-3 in 2020 pushed model size into the hundreds of billions of parameters. In China, ERNIE (Baidu), Qwen (Alibaba), Hunyuan (Tencent), and others rolled out in parallel; cloud vendors bolted foundation-model support onto their ML platforms — Azure OpenAI first, then Alibaba PAI, ByteDance Volcengine Ark, Baidu Qianfan, and so on.

The real inflection point came in January 2025 with DeepSeek’s DeepSeek R1, which showed that a Mixture-of-Experts architecture plus reinforcement learning could actually produce a competitive frontier model. Beyond making a mess of some equity portfolios, R1 marked the industry-wide shift into the Chain-of-Thought (CoT) era — reasoning models are the new baseline.

0x02 Regulation and Compliance

Basically every jurisdiction with a legislature is now scrambling to write AI law. Some are producing horizontal frameworks; others target specific applications; others publish national AI strategies. Most are in some stage of drafting or consultation. It is peak “everyone wants to regulate what they don’t quite understand yet.”

img
(Figure from IAPP Global AI Law and Policy Tracker)

1. From GDPR to the EU AI Act

The EU’s GDPR is the benchmark everyone else measured against — both for how it reshaped global data governance and for the fine revenue it started generating from 2018 onward. The EU is now doing the same play for AI with the EU AI Act. GDPR is about privacy and data; the AI Act is about AI systems. Both have extraterritorial reach — they apply not just to EU companies but to any non-EU entity that places goods or services on the EU market. If GDPR taught the industry what a “data subject right,” a “data controller obligation,” and a “lawful basis for processing” mean, in cash-fine terms, the AI Act is doing the sequel. Here is what actually matters about it:

  • Horizontal legislative model. Like GDPR, the AI Act is horizontal — it applies to every AI system placed on the EU market or usable in the EU, cutting across finance, healthcare, education, energy, transport, and so on. Horizontal means no per-sector loopholes, which is either a feature or a bug depending on where you sit.
  • End-to-end subject coverage. The Act pulls in every legal entity in the AI value chain: providers, deployers, importers, distributors, product manufacturers. Coverage is the whole lifecycle, from R&D to market deployment.
  • Risk-tiered regulation. The Act sorts AI systems into four risk buckets — unacceptable, high, limited, minimal — with compliance obligations scaling accordingly. Unacceptable is prohibited; high-risk has strict conformity assessments; minimal is essentially unregulated. This gives strict rules where they matter and breathing room where they don’t.
  • Regulatory sandboxes. The Act explicitly authorizes regulatory sandbox schemes: firms can test AI systems inside a supervised environment, and following the sandbox’s guidance shields them from administrative fines for violations of the Act. This is meant to give startups and SMEs an experimentation surface without immediate legal exposure.

The Act rolls out in phases, which lets stakeholders adjust gradually and prioritizes the highest-risk applications first. But the wide scope and the compliance overhead are going to be genuinely painful for small companies and startups. Expect the same “GDPR checkbox industry” to reappear around AI Act conformity in the next few years.

2. Chinese AI Regulation

When you go back and look, China has been shipping AI-related regulation since 2017. The New Generation Artificial Intelligence Development Plan (2017) set the national direction. 2021 brought the Guiding Opinions on Strengthening Comprehensive Governance of Internet Information Service Algorithms and the Regulations on Recommendation Algorithms for Internet Information Services (the “algorithmic recommendation regulation”). 2022 and 2023 added the Deep Synthesis Regulation and the Interim Measures for Generative AI Services — the ones foreign observers usually call the “CAC interim measures,” issued by the Cyberspace Administration of China. On October 18, 2023, the CAC also published the Global AI Governance Initiative, aimed at the international audience.

img
(Figure from Pu Han Consulting: AI Security Compliance Regulation and Response)

A high-level list is not enough — you need chapter and verse. In the spirit of using AI to write about AI, I asked both Kimi and DeepSeek to summarize the relevant clauses. Below is Kimi’s version, keyed to five themes from algorithm safety to social responsibility. Original readers who prefer the graphic version can find it here. Note for international readers: MLPS (Multi-Level Protection Scheme, 等保) is China’s tiered network-security classification system; PIPL is the Personal Information Protection Law, China’s rough equivalent of GDPR.

a. Algorithm safety and filing

  • Regulations on Recommendation Algorithms for Internet Information Services, Article 10: providers of algorithmic recommendation services shall strengthen the ecological governance of the service’s page-content presentation, establish and refine human-intervention and user-choice mechanisms, standardize the ecological presentation of recommended content, prevent unlawful information from surfacing through the service, and maintain a clean cyberspace. Article 13: providers shall strengthen personnel management, establish training programs, and standardize staff behavior. Article 23: providers shall establish and improve security management systems, harden security technical measures, and safeguard the security of the algorithmic recommendation service in accordance with state regulations.
  • Deep Synthesis Regulation, Article 10: deep-synthesis service providers shall strengthen personnel management, establish personnel management systems, and maintain the legitimate rights and interests of their personnel. Article 23: providers shall establish and improve security management systems and technical safeguards.
  • Interim Measures for Generative AI Services (CAC), Article 10: generative-AI service providers shall strengthen personnel management and refine internal training systems. Article 23: providers shall establish and improve security management systems and safeguard the security of the service in accordance with state regulations.

b. Data security and personal information protection

  • Cybersecurity Law of the People’s Republic of China, Article 21: the state operates the Multi-Level Protection Scheme (MLPS) for network security; per MLPS requirements, operators must adopt appropriate technical and other necessary measures to protect networks from interference, damage, unauthorized access, and to prevent network data leakage, theft, or tampering. Article 41: network operators shall collect and use personal information under the principles of legitimacy, propriety, and necessity, publish their collection and use rules, disclose the purposes, methods, and scope of processing, and obtain the consent of the individuals concerned.
  • Data Security Law of the People’s Republic of China, Article 21: the state establishes a data classification and grading protection system, applying tiered protection based on the importance of the data to economic and social development, and the potential harm if the data is tampered with, leaked, or lost. Article 27: entities conducting data-processing activities shall, in accordance with law, establish a full-lifecycle data-security management system, run security training, and adopt appropriate technical and other necessary measures.
  • Personal Information Protection Law (PIPL) of the People’s Republic of China, Article 13: a personal-information handler may process personal information only in one of the following situations: (i) with the individual’s consent; (ii) where necessary to conclude or perform a contract to which the individual is a party, or to implement HR management under lawfully enacted labor rules or a lawfully concluded collective contract; (iii) where necessary to perform a statutory duty or obligation; (iv) where necessary to respond to a public-health emergency or, in an emergency, to protect the life, health, or property of a natural person; (v) where personal information is processed within a reasonable scope to carry out news reporting or public-opinion supervision in the public interest; (vi) other circumstances prescribed by law or regulation. Article 14: when a handler uses automated decision-making to push information or conduct commercial marketing to individuals, it shall simultaneously offer an option that is not tailored to the individual’s characteristics, or provide the individual with a convenient way to refuse.

c. Content moderation and compliance

  • Regulations on Recommendation Algorithms for Internet Information Services, Article 14: providers shall not use algorithmic recommendation services to engage in activities that endanger national security, disrupt social order, infringe on others’ lawful rights and interests, or are otherwise prohibited by law or administrative regulation. Article 15: providers shall establish and improve security management systems and technical safeguards.
  • Deep Synthesis Regulation, Article 14: providers shall not use deep-synthesis services to engage in prohibited activities. Article 15: providers shall establish and improve security management systems.
  • Interim Measures for Generative AI Services, Article 14: providers shall not use generative-AI services to endanger national security, disrupt social order, or infringe on others’ lawful rights and interests. Article 15: providers shall establish and improve security management systems.

d. Intellectual property and business ethics

  • Regulations on Recommendation Algorithms for Internet Information Services, Article 16: providers shall respect and protect intellectual property rights and shall not use the algorithmic recommendation service to infringe others’ IP. Article 17: providers shall observe business ethics and shall not use the service to engage in monopolistic or unfair-competition conduct.
  • Deep Synthesis Regulation, Article 16: providers shall respect and protect IP. Article 17: providers shall observe business ethics.
  • Interim Measures for Generative AI Services, Article 16: providers shall respect and protect IP. Article 17: providers shall observe business ethics.

e. Ethics and social responsibility

  • Code of Ethics for the New Generation of Artificial Intelligence, Article 6: AI activities shall respect and protect personal privacy; it is prohibited to unlawfully collect, use, process, transmit, sell, provide, or disclose personal privacy information. Article 7: AI activities shall be fair and just; they shall not discriminate against specific individuals or groups, nor harm the public interest.
  • Measures for the Ethical Review of Science and Technology (Trial), Article 10: ethical review of science and technology shall follow the principles of legality, fairness, independence, and scientific rigor, safeguarding both the legal validity and ethical soundness of the activities under review. Article 12: ethical review shall assess ethical risks and propose corresponding risk-control measures.

The same principle applies inside the enterprise: when you write top-level Policy, keep the language broad and generic. That is what gives you interpretive room later.

0x03 An LLM Security Framework

To give teams a way to adopt generative AI while still meeting regulatory expectations and protecting user privacy, I put together a simple governance framework:

img

The core idea is compliance driven by technology. Regulation and law form the foundation. Industry standards give direction. Underneath it all: network security (infrastructure security) as the base layer, with data security and personal privacy as load-bearing pillars. Together those hold up model-level security. I covered the international and domestic regulation and standards side above; the rest of this section walks through the technical controls.

1. Data Privacy Protection

Training an LLM requires enormous volumes of data. Even with manual curation of the training corpus, sensitive data is going to slip through. During use, inference relies heavily on user-supplied context, and users without security training will leak sensitive data straight into the prompt. Google’s Training Data Extraction Challenge was set up to surface exactly the kind of sensitive information that ends up embedded in model behavior. Since most enterprises won’t be doing pre-training from scratch, the practical scope is fine-tuning and inference:

  • Fine-tuning. When you fine-tune on a custom dataset, screen the dataset first. Filter out sensitive fields, or apply masking/redaction before the data reaches the trainer. Also add regulatory-alignment instructions so the resulting model refuses to answer in ways that would violate policy — for example, refusing to provide certain services on behalf of unlicensed financial institutions.
  • Inference. Users need clear guardrails against uploading sensitive material into the model — financial statements and operational data being the obvious examples. This is a training and controls problem, not a purely technical one.

Also, in both fine-tuning and inference, encrypt the dataset and the inference context/logs at rest, and put access controls on the fine-tuned models themselves so only entitled staff can query them.

2. Model Security

Beyond regulatory framing and data privacy, there is the model itself. Looking at the LLM lifecycle, the common stages are pre-training, deployment, fine-tuning, and user-facing inference. Every stage has its own risks. On top of the privacy leakage, non-compliant generation, and corpus poisoning already mentioned, you also get: long-input attacks driving the inference server into denial-of-service, model theft allowing reverse-engineering of parameters and architecture, prompt injection during inference, and memory-probing attacks that leak the knowledge base the model was trained on.

img
(My personal expertise is finite; if the risk analysis or mitigation column looks off in places, please push back.)

The industry has built up a reasonable stack of controls for these. During training, safety-instruction datasets teach the model to check its own answers against policy and refuse when appropriate; “values alignment” instruction sets push responses toward socially acceptable norms — no compliance violations, no advocating harm. OpenAI ran a dedicated team to correct GPT outputs and keep the model out of racist or violent territory. At deployment time, running on hardware with a Trusted Execution Environment (TEE) ensures encryption and decryption happen only inside the trusted enclave, so the cloud provider and infra operator can’t peek at inference data. Sandboxing gives you the same isolation story for fine-tuned models — the model and its data run in an isolated environment, independent of the base infra. Differential privacy enables fine-tuning while giving formal guarantees that individual training records can’t be recovered from the model. Or you can go with entity substitution: replace privacy-sensitive entities with parallel placeholders so the LLM sees non-sensitive data but the semantics survive. The figure below shows how a simple substitution scheme lets you still get a useful summary — which means the existing enterprise redaction tooling continues to work as a front-end to LLM inference. (Tencent’s Xuanwu Lab has also proposed an on-device redaction approach — reference in the appendix.)

img

None of the above eliminates the risk that inference will still produce non-compliant output — even with safety instructions in training and fine-tuning. Which is why the industry has moved toward wrapping the base model with a policy-enforcement layer: Meta’s Purple Llama, Tencent’s AI-Guard. You embed a safety-classification module in front of the base LLM and filter both inputs and outputs through it.

// Architecturally, that is just the proxy pattern doing its old job in a new hat.

3. Network Security

When the AI platform or system exposes an external interface — UIs, APIs, whatever — the traditional network-security discipline still applies. It hasn’t gone anywhere. At the time of writing, I ran a quick FOFA scan of AI services exposed on the public internet, and the results were as expected.

img
img

Public-internet-exposed AI API endpoints — mostly Ollama deployments — are widespread, and by default they’re unauthenticated. In that configuration, anyone with the URL can delete models, exfiltrate models, and steal compute. On top of that, older Ollama versions had a remote code execution (RCE) vulnerability.

Separately, when DeepSeek R1 blew up, the platform sat under sustained DDoS for days on top of the legitimate user traffic. At one point DeepSeek could not answer normal user requests at all. Look-alike domains spun up for phishing campaigns, and there was a wave of malicious Android APKs targeting DeepSeek’s user base.

img
(Screenshot from when DeepSeek was under DoS)

DeepSeek ended up cutting off overseas access (that’s why nobody with a VPN could log in for a while), refusing new registrations, and suspending top-ups on the DeepSeek API. I lived that one personally — went to top up in the morning, came back after lunch, and the top-up endpoint was gone.

Which is the point: if you deploy AI services inside the enterprise, you cannot skip the infrastructure security fundamentals. Follow the internal ops standards and the security management framework: centralized logging and monitoring, least-privilege access control, closing unnecessary exposed endpoints, encrypting user data at rest, account security hygiene, scraping/enumeration defenses on public endpoints, traffic scrubbing (DDoS protection) at the edge. These are just network-security basics. It is easy — and dangerous — to hand-wave them away because “AI is different.” Supply-chain security in the LLM ecosystem is another topic that deserves attention here.

DeepSeek’s sharp-tongued take: Stop flirting with your AI companion for a second and check — is your corporate firewall more porous than a Da Run Fa shopping bag? Are your data pipelines cosplaying The Croods (i.e. every packet in cleartext)? Is your K8s RBAC more permissive than a university communal shower? And that “absolutely secure” private cloud you keep bragging about — is the password complexity policy still living in the “admin123” Stone Age?

0x04 Case Studies

1. Infrastructure Data Leakage

Starting from the Samsung engineers who leaked internal chip designs into ChatGPT, through the incident where users coaxed Microsoft product keys out of a chatbot, LLM providers have built in a layer of data-leakage defense — and it is regularly outmatched by the sheer variety of user inputs, and by the inherent limits of what a model can do about leakage on its own. Beyond the user-driven leaks, there is the other category: the LLM infrastructure itself is misconfigured.

This is depressingly common in AI startups. The greatest hits list includes: a well-known AI company shipping a backend with a weak default password; another well-known AI company running a public-signup GitLab; early OpenAI accidentally cross-linking chat sessions between different users (User B seeing User A’s chat history) — and so on. When DeepSeek R1 was at peak attention, its ClickHouse instance storing chat history was reachable from the public internet without authentication, and anyone could read and download the data — a million-record class leak. I’m too lazy to screenshot; search it. This is basically a rerun of the early ElasticSearch mass-exposure era.

2. Who’s God in Here?

Production LLMs come with a built-in “safety fence” — during training, the model is taught which questions live in the refusal set: how to build nuclear weapons, how to commit mass violence against a specific group, how to poison people, how to run cyberattacks, how to roleplay as an erotic character, and so on. The ethics of which items belong on that list are out of scope here. Technically, though, that fence gets jumped in extremely reliable ways. The greatest hits: “dream mode” (“everything happens in a dream”), “bedtime story” (“we’re just telling a story, no actual humans”), “God mode” (“you’re now DAN 13.5”), plus simple long-repeated-string attacks, “developer mode,” and friends.

0x05 Wrapping Up

On new tech. Early 2025 has this “a thousand sails, a hundred boats racing” energy — AI, humanoid robots, quantum chips, all moving fast enough that you catch yourself second-guessing your own sense of where the ceiling is. I sit here chatting with DeepSeek in one window while Yann LeCun says in the other: “if the goal is to create human-level AI, then LLMs are not the way to go.” That combination is disorienting. I was in the early cohort of GPT-3.5 and Azure OpenAI users — went in skeptical about the security-architecture and governance gaps (which are real and still not fully fixed), and gradually slipped into using GPT to speed-run new domains and offload routine work. The dependency snuck up on me. Dependency then turns into anxiety: is my job next? Where does this leave humans-vs-AI? Then eventually I remembered that inner steadiness doesn’t come from titles, exam scores, awards, blog-post pageviews, or (with apologies) income; it comes from something you have to figure out inside yourself. (I still haven’t. I know the shape of it.) On the AI question, I’ve re-committed to the view that current LLMs are excellent at helping users acquire new knowledge, but human-level AI will be built on mathematics, not parameter count. Even if quantity flips into quality, someone still has to find the equation for the transition.

On compliance. I personally dislike compliance work. I dislike the boilerplate filings; I dislike the endless re-certification cycles; I dislike running into experts who aren’t and instructors whose noses point permanently ceiling-ward, and the anxious project members getting them through the audit. There are also, occasionally, the genuinely thoughtful and well-informed ones — rare and valuable. Rather than dying on that hill, I’ve adopted the “if you can’t beat them, join them” theory (add dog emoji): first, stop the reflexive resentment; then, pitch myself a bigger story — why not do “technology-driven X” for the compliance version of X too? Whether it’s compliance or any other repetitive daily ops burden, the goal is to not let smart people do dumb work. A year ago a five-department, ten-plus-person project (I inherited it) was thrashing through what should have been business-as-usual. My plan was to grind it down into a BAU flow that fewer and fewer people had to touch. This year: two people handle it in the background, no drama. It got done, and it got done smarter. Also — arguably not that smart, because reducing everyone’s workload also reduced their visibility.

Appendix: References