Murphy’s Law doesn’t say bad things will happen — it says if something can go wrong, it will. ——Interstellar

0x00 Intro

Lately I’ve been through some pretty sloppy design and operations work. Sure, the world is basically one giant ragtag outfit, and I’ve had friends cheer me up and colleagues talk me through it — after the initial frustration, I ended up picking up some useful lessons. If I don’t write them down, I’ll have missed a real chance to grow. So I figured I’d come at security operations from an architecture angle and share what I’ve learned. (Note: “security operations” here doesn’t just mean SOC — it means the day-to-day work of the entire information security team.)

0x01 Design for Failure

In a proper company of any reasonable size, security operations typically comes with three lines of defense from outside to inside. Top-down you’ve got policies → standards → procedures. In the middle: alerts → incidents → tickets. At the bottom: response → forensics → post-mortem. There are systems up front and platforms in the back, data on one side and scripts on the other. The security team tends to think it’s got everything covered, running at full capacity. So why do so many problems still slip through? In most cases, it comes down to a lack of validation — and a lack of people qualified to actually do the validating.

To be clear, this has nothing to do with security validation products. The business complexity inside most companies isn’t quite at the point where commercial security validation tools can fully deliver value. What I want to do instead is walk through this from a “design for failure” mindset — not a new concept by any means, it’s been baked into systems engineering for ages. I’m just borrowing it for operations. The core idea isn’t really about design at all. It’s about accepting and internalizing, from day one, that failure is inevitable. As Murphy’s Law says: if it can happen, it will. And since there’s no such thing as an absolutely secure system, problems are guaranteed to show up:

You think a dedicated leased line solves everything — turns out both endpoints have critical vulnerabilities. You think all the security products are properly deployed — turns out HA is just decorative, and the security system itself has a knock backdoor. You think you’re buying a “cutting-edge” product — it might just be a China-market special with stripped features. You think IP allowlisting makes you a fortress — sure, daily operations got more painful, but what you don’t know is that every backend system runs on weak passwords that anyone can guess. You think SSO integration is the final answer — but somehow there’s not even MFA. You think MFA covers you — but the MFA tokens have no expiry. You think you’ve raised the weakest link high enough — then one phishing email wipes out everything. And that’s before you count employees who click the phish AND walk away without locking their screen.
You think the key went through multiple KSO-managed generation and transport steps — but the receiver just dropped it in a config file (so much for treating keys and secrets with equal care). You think EaaS is providing stable encrypt/decrypt services — but the receiver is just doing the computation locally. You think your data classification is solid — but the logs are leaking PII all over the place, and persisted data was never sanitized. You think CAPTCHA is blocking bot traffic — but the config is still in observation mode.
You think endpoints are secure — then you find out someone forgot to revoke admin rights after closing a ticket. You think your control policies are airtight — but someone with a privileged account has been quietly installing all kinds of software outside the allowlist. You think DLP is precise and web activity is properly controlled — but a real data exfiltration only needs to happen once.
You think your scanner rules are comprehensive — but a third-party library slipped through. You think the security section you added to the architecture review actually gets followed — but nobody’s doing it. You think patching one vulnerability is done — but the patch itself introduced a new one. You think the training worked — but half the audience was asleep. You think SIT/UAT/PROD are properly isolated — but someone already bridged them together somewhere. You think TLS client certificates are a great auth mechanism — but the server isn’t actually validating the CA or the expiry.
You think the vendor is experienced and professional — but their policies are empty, and they tell you the job’s done. You think one policy rule will trigger alerts — but months pass without a single one. You think you’ve covered tons of metrics and built a bunch of dashboards — but when an incident actually happens, the data is one-dimensional and useless for analysis. You think “proficient in threat hunting and forensics” on the resume means real incident experience — turns out they just finished a Splunk beginner tutorial. You think five years of senior operations experience means expertise — but mostly it means watching alerts scroll by.
You think the tabletop exercise was well-prepared and impressively executed — but it only “succeeded” because of a lucky accident that made two failures cancel out. You think audits will surface problems — but getting meaningful feedback from an audit is harder than explaining music theory to a cow. You think you’re experienced, theory-solid, practice-tested — and you might still be blind to your own blind spots. (Yes, that last one’s aimed at me.)

The thing is, in day-to-day operations, most people can’t see the forest for the trees. They think they’ve got their own piece locked down, overestimate how professional they are, and carry an Emperor’s New Clothes-style confidence. Reality is usually one of two things: either there’s no real design (even though everyone thinks there is), or there’s way too much design (hey, look how thorough we are!). The first case isn’t worth discussing. The second case loads everything up with complex metrics, multiple processes, and procedures. And ironically, those very processes and metrics — regardless of whether they’re reasonable — create an illusion for everyone involved: “We/the system are secure!” A new kind of blind spot forms right under the spotlight.

I once actually ran into an approval workflow with 6 nodes and a 6-month validity period after approval. But the only thing it did was let you contact an IT person to add you to the right group. (Look how secure!)

Back to designing for failure: from an architecture perspective, it’s not just about resource-level redundancy and data backups. It’s also about making applications modular so they can gracefully degrade or self-heal. On the operations side (the operations module within architectural design), you also need solid change management, test and validation, disaster drills, backup and recovery, and so on.

Looking at BAU through a design-for-failure lens, it breaks down into two areas (both honestly well-worn topics):

Effectiveness validation:

Regular risk assessments — don’t trust any existing allowlist, don’t trust any system’s claimed stability. Assess periodically, monitor continuously.
Ensure playbooks exist for as many known scenarios as possible — you can’t guarantee the right person will be available when an incident hits. Everyone needs to be able to work from an SOP.
Verify that redundancy and backups actually work — is the backup data empty? Did the UPS battery drain? Can you actually import the key backup?
Verify that monitoring is working in real time — are alerts firing? Are logs flowing? Can you log into the monitoring system? Did you discover during an investigation that a component switch was never turned on?
Test after training — you’re delivering training while the remote folks are napping and the in-person folks are chatting.

Continuity validation:

Run quarterly drills — not to manufacture problems for show, but to keep security awareness alive.
Write a proper BCP for the security team — even without the butcher, you can’t eat the pig with the hair still on.
Invest in automation — machines are more reliable than humans for repetitive operations. Humans should be focused on making sure the automation is actually running.
Modularize processes, but resist modularizing people — in architecture design, design for failure means modular components; in operations, you need every person’s module to have a backup and cross-functional support. “I only do X, I only know X” doesn’t cut it. (But you’ll need to pay for it.)

Even with all that in place, surprises still happen. Sometimes from an unexpected action by a vendor (the world really is one giant ragtag outfit). Sometimes because nothing was documented and you have to prove your own innocence. But circling back to the main point: the heart of designing for failure is accepting failure. Incidents are unpredictable — especially in the middle of incident response, the most important thing is staying calm, proposing a plan, pulling in the right teams, and getting remediation done fast. Figuring out who’s accountable can wait. (Some people may disagree — maybe they’re already thinking about post-incident blame before the fire is even out.)

0x02 Security as a Service

Security work is inherently asymmetric. To maintain confidentiality, you have to sacrifice transparency. To enforce secure defaults, you have to intrude on the architecture. To do SDLC properly, you have to hook into business processes. If one side needs zero business friction, the other side needs infrastructure rework. Externally, you’re delivering security as a service. Internally, you’re building a security platform. And delivering security externally is really the process of turning BAU into processes, turning processes into platforms, and turning platforms into API interfaces. That’s a bit different from SaaS — SaaS delivers a product to customers, this is delivering a service made into a product. But then again, isn’t the enterprise security team basically an internal vendor?

For any company, whether the dev team or the security team is building the security platform, maintaining security service continuity is essentially the company paying the bill for failure. The cost of design and planning, the cost of development and implementation, maintenance costs, training costs — all of it. And when the company changes direction, it inevitably has to pay again for the new target.

The reason I’m even talking about security as a service is that I recently went through some vendor decks on “Security Operations as a Service.” Pretty much all of them approached it from two angles: lifecycle management and SOC. Sprinkled throughout were emerging technologies and products — AI applications, SOAR optimization, that kind of thing. But reading the whole thing, I couldn’t find the actual operations in “security operations as a service.” On one hand it looked like it could wrap up an entire SOC operation. But something was consistently missing.

“X as a service” isn’t a new idea, but I couldn’t help wondering: is this just a response to a cold market and rising labor costs? A way to use a staffing pool model to cut costs for clients while squeezing maximum value out of security operations workers in the middle? It reminded me of the human-chain computers in The Three-Body Problem. Large enterprises that can afford a full SOC team don’t care about these marginal savings. Small companies that can’t afford one don’t care about security in the first place. Security Operations as a Service seems to be gradually turning security operations into pure labor-intensive work. Worth noting: cloud vendors’ security products with Pay As You Go pricing are actually closer to the real spirit of security as a service. Of course, from a cost perspective, vendor pitches may be full of fireworks — but the enterprise buyers are running lean. In an era where your position determines your perspective, it’s worth staying at least a little objective.

0x03 Wrap Up

When I sat down to write this piece on security operations, I realized something. However much I used to hate BAU work back in the day, that’s exactly how seriously I need to examine it now. Wanting to see both the trees and the forest at the same time — that’s genuinely hard. On one hand, BAU is unavoidable. On the other hand, you have to let go of your fixed assumptions about technology. Because not everything that holds up logically will actually run as expected in a real operational context. Especially in financial services — the more foundational the thing, the more likely it is to defy common sense. There are so many unique scenarios.

Like, you know perfectly well that proper auth on a public-facing API is more secure than a dedicated line with IP allowlisting — but the default answer is always allowlisting, leased lines, and hardcoded values. Over time, everyone there starts believing that ops-dev separation combined with dedicated line IP allowlists is “very secure.” They’ll say “defense in depth” out loud but keep doing the same old things, occasionally adding a trendy label like “chaos engineering” — what that actually looks like in practice is anyone’s guess. The gap between traditional financial architecture and internet-native financial architecture is huge. You almost can’t trust the design docs at all — you have to look at what actually got built. Because in traditional financial systems, some core transaction systems don’t even use DNS, since DNS might introduce caching. (Whether that’s a valid reason is debatable.)

With large language models taking off, I’m actually somewhat pessimistic about the future for security operations practitioners. Take GPT-4 as a reference: it can write glue code (Doubao and Tongyi can’t even match GPT-3.5 for coding), do code auditing — somewhere between a junior and senior engineer in efficiency. Gradually, the three lines of defense will evolve into: one senior person writes the prompts, junior engineers operate alongside GPT, costs drop, and suddenly “Security Operations as a Service” becomes real.

There was a point when I kept going back and forth thinking about the difference between -1 to 0 and 0 to 1. In architecture design, there’s actually not much difference: if you have no architecture, you design one from scratch (V1). If you have a broken architecture (T-1), you design a proper one (V1) and implement it in N phases. But in operations, -1 to 0 and 0 to 1 are worlds apart. Operations can’t be fixed in one jump. Setting aside optimization — just “straightening out the mess” is a gap that might not be crossable. The labor cost alone is like first reining in a horse running the wrong direction before you can ride it forward. Trying to go from 0 to 1 in operations while still under T-1 conditions — the pain of that speaks for itself.