Written on July 19th, published on July 25th

1. Operations and SOC

There’s already plenty written about security operations, and many enterprises entered the operations era years ago, sooner or later. Thanks to my experience going through several 0-1 security builds, this post looks back at SOC design from a team and architecture perspective. My experience is limited, so please go easy on me, operations veterans.

Also, you can read the following with some questions in mind. For instance: you bought a lot of products, software, and equipment—are you actually using them? When do you need to establish/split out a SOC team? Should security platform engineers and security operations engineers cross-train? You’ve defined a bunch of operational metrics—do they really reflect operational quality? Engineers wrote dozens or hundreds of rules—which ones actually got triggered?

1.1 Team Collaboration

Basically, after the security team builds out the infrastructure, it enters the operations phase, and then the operations team gradually scales up. The security lead quickly gets buried under all kinds of dashboards. As for whether they can understand and measure the quality of the operations team, we’ll get to that later. First, let’s look at what a SOC team roughly looks like. From a response workflow perspective, you typically have Tier 1 to Tier 3. From a responsibility standpoint, large SOC teams generally handle monitoring operations, red-blue team exercises, threat intelligence, and forensics and incident response. Whether the team is big enough depends on whether the SOC leader is a Manager or a Director.

From a capability maturity model perspective CMM-SOC, a SOC that can operate normally needs at least Level 2 characteristics. Big companies’ SOCs are often already at Level 3 (at least in their marketing materials they claim to be at Level 3 and above—I remember an Alibaba security expert previously proposed the concept of data-intelligence-driven security operations), and they have 7*24 oncall capability and can coordinate with NOC to handle issues.

As for collaboration workflows, I’m still using the diagram from my previous blog post. The systems here are generally maintained by security ops for stability, platform engineers develop new systems or integrate systems, data warehouse construction comes from risk control or data center teams, and security operations needs incident workflows, monitoring and alerting, etc. Along with corresponding security BPs serving as points of contact to quickly handle issues on business lines.

Of course, this depends on enterprise organizational structure—different structures coordinate differently. Overall, there are dedicated teams maintaining platforms and products, letting operations focus on automated, scenario-based incident detection. Some SOC teams might just be alert operations teams, which are highly replaceable and usually consist of vendor staff stationed on-site for the client.

1.2 Data Platform

The data computing platform is crucial to the operations process—most security operations engineers’ work should be done here. By doing Batch Ingestion on Data Sources and storing them in a Data Lake, cloud-based Blob Storage can now handle the data lake role. Taking AWS as an example, you can use Redshift or S3. After that, data gets cleaned, usually within a Pipeline that completes cleaning, normalization, and validation. Worth noting here is that the normalization process needs a data dictionary to uniformly map different fields to preset fields (establishing a standard data dictionary also takes effort).

After that, you can write rules. Besides applying to individual systems, rules are most produced on the data computing platform—we’ll talk about metrics for measuring operations later. Spark with Groovy works fine, as does Splunk’s built-in SPL. Most of these are condition-based rules—if you want to use machine learning methods, you need another key process: feature engineering (actually, aside from commercial products’ built-in machine learning, I haven’t seen machine learning/deep learning being used for intrusion detection outside of big companies). Feature engineering can simply be understood as a coefficient set for a polynomial function. Feature mining can be done through expert experience or by having neural networks automatically search. I haven’t paid attention to this area for a while—checking my old feature engineering notes, they’re from 2018. Finally, after training & evaluating the model, it gets deployed and provides API interfaces. On one hand, you can consume the model through APIs; on the other, you can complete visualization and automation. The data computing platform can also introduce a lot of external data, like purchased Threat Intelligence. Splunk Enterprise Security currently supports similar functionality. Note that TI is charged per query count, so you can appropriately cache list data to avoid duplicate queries within 24h. And provide it to other systems. (BTW: Virustotal engineers mocked a certain vendor for stealing their data).

Note: I don’t have practical architecture experience implementing data warehouses with Hadoop as the underlying layer. At work, I’ve only used ODPS, SLS, Splunk, Groovy, and Spark. Cloud solutions I’ve only done POCs for AWS and Azure, and don’t know the specific capacity and performance in production environments. (Theoretically, you just need to increase funding on the cloud to achieve specific performance—the architecture itself isn’t affected)

1.3 Operational Quality

Assuming we take data-driven security operations as our principle and make automation, standardization, scenario-based, and visualization our goals. (Often what you lack is what you emphasize most 😓)

Process
We use the IPDRR model to represent the process and map security operations content into it. As shown in the diagram, security operations in the identify phase mainly does asset management and risk assessment (these assessment standards need to be provided by the governance or architecture team). The protect phase involves implementing security controls, developing rules, etc. Similarly, in subsequent phases, real-time monitoring and alerting, incident management, disaster recovery, etc.

We can also switch perspectives, like looking at IPDRR from a Detect angle. The Identify process is actually doing Threat Modeling, the Protect process involves some Rule Tuning work (equipment and services are provided by security ops or security platform teams), and so on.
Metrics
To measure operational quality, you need metrics for assessment. In the above process, besides MTTA, MTTD, MTTR, you also need to focus on the following entity metrics to measure SOC operational quality.
You can see there are entities like alerts, incidents, SOPs, rules, Playbooks, etc. Here we focus on entities other than System (System has other metrics—here we only focus on SOC operations metrics). For example: Volume (total, new additions), Accuracy (T-P, F-P, T-N, F-N), Priority (P0 incidents - PN incidents), Cost (time/development time, forensics time, etc., manpower).
In this process, you can consider each entity’s accuracy rate. For instance, how many alerts are correct, whether Playbooks executed correctly, etc. The total and new additions for each (once this data is fed into the data platform, you just need to extract the corresponding reports with daily, weekly, monthly, quarterly views. When reporting, leadership doesn’t care about details—in fact, aside from the operations team itself, other teams care more about quality than process). Besides this, you also need to pay attention to companion metrics and opposing metrics. For example, when considering coverage rate, you need to consider health as a companion metric. Otherwise, you might have 100% coverage but 80% health (the other 20% might be repeatedly restarting due to CPU/MEM, or unable to execute processes normally, etc.), similar to training completion rate and phishing click rate. Opposing metrics include T-P and F-P—you need to look at both the proportion of actual problems correctly triggering alerts and the proportion of alerts triggered when there are no actual problems. Besides this, you’ve surely heard of many frameworks—for example, when discussing SOC architecture design internally. I’ve also heard voices saying there are too many frameworks to choose from, like SOC already uses the ATT&CK framework. Actually, there’s no conflict here. SOC uses ATT&CK to measure detection scenario coverage on the data platform—ATT&CK is precisely a reference framework for scenario-based goals. But this raises a new question: you referenced ATT&CK to cover 20 scenarios and deployed 70 rules—are they really effective? How do you measure? Do you trigger detection on the computing platform or sync rules to each System to execute? Similarly, there’s purchased threat intelligence. We know threat intelligence can provide IOCs like domain, ip, url, cert, ja3, email, hash, and these IOCs can also be applied to different systems like SIEM, ASM, WAF, etc. Here we need new metrics. Like the number of triggered scenarios, the number of alerts generated by threat intelligence (hit rate), the accuracy rate of these alerts, which scenarios they cover.
Finally, you need some statistical dimension metrics, like: What rules do the Top 10 alerts come from, which Playbooks they triggered respectively (and how many times executed), which stages of the KillChain they hit, which ATT&CK scenarios they hit, what categories the incidents are, and which new types of incidents were converted into SOPs. This measures the SOC team’s operational quality. You need to automate the main threat attack defenses to invest more energy in high-priority incidents.

Going back to the goals mentioned at the beginning of this section, we can select appropriate metrics to measure the progress of scenario-based and automation. Also, I have to mention Microsoft’s products—when you push a configuration from the management console, the standard answer from technical support is often: takes 24h+ to take effect, depending on user count. But actually sometimes it’s around 6h, sometimes it’s over 24h. Especially after you do SOD, if handling requires activating privileged accounts, it might take about 10 minutes to take effect after activating permissions once, sometimes even requiring login/logout to refresh and see the corresponding management interface.

2. Collective Intelligence

I first learned about the term Collective Intelligence from the book “Programming Collective Intelligence.” But this term really feels both strange and familiar to me. Though I’ve heard it many times, I’ve rarely seen it in reality. Summarizing, it seems on one hand, different individual levels lead to varying degrees of arrogance and prejudice, while on the other hand, group collaboration also brings a certain degree of chaos. I believe every enterprise has some engineers who are “born at the wrong time”—on one side thinking they’re thoroughbreds waiting to meet Bole, on the other side regretting that no one in the company has the insight to recognize their talent. But when they actually benefit from collective intelligence, they often ignore the platform’s role. Those colleagues whose work hasn’t met their Bole have gradually built up an emotional filter toward work over time. Continuous negative feedback (no raise, no promotion, no projects, getting challenged) makes them increasingly resistant to input from the opposing side. Even if they’re professional in a certain direction, they’ve lost the initiative to promote their own solutions. This is unfavorable for the team. Because improving individual intelligence doesn’t necessarily benefit collective intelligence—on the contrary, it lowers the overall level.

Looking from four aspects, the keys to generating collective intelligence are information sharing, collective decision-making, communication, innovation. Mainly manifested as collaboration and decision-making. And your collaboration and decision-making can only be based on your level of understanding. This brings up another concept—comprehensible input and comprehensible output. Because collective intelligence requires individual participation, information sharing and communication are key to collective collaboration. Comprehensible input can help individuals actively learn and improve themselves, comprehensible output can help individuals promote themselves.

Taking SOC operations as an example, undoubtedly, the establishment and operation of SOC relies on massive collaboration. Internally, you need collaboration among tier 1, tier 2, tier 3 colleagues; externally, you need coordination with risk control, compliance, data, and other departments. It’s not hard to find that incident response, accident review, reporting, writing documentation, training, and other daily work all require extensive communication and coordination—almost everywhere involves collective decision-making. It also involves the balance between individual and collective decisions. We can certainly use expert experience or leadership (hardline approach) to make decisions, but we can also use collective intelligence. Many times when facing decisions from leadership, can you use collective intelligence to make up for individual intelligence’s shortcomings? (Does your collective have exceptionally outstanding individual intelligence?). Let’s return to the concept of comprehensible input and output. Suppose when a Manager proposes a solution that needs to be implemented, you might have a better solution, but you can’t just outright deny their solution or tell them it can’t be done or is very difficult. If the other party hasn’t received professional manager training, at this point they’ve received incomprehensible input and will definitely be extremely resistant internally. When you propose a new solution again, you’ll inevitably encounter obstacles. If it also involves knowledge they haven’t yet encountered, it’ll be even harder to promote the solution. You can provide choices, let the Manager choose how to implement, gradually lean toward the new solution during the selection process, and finally provide it. This is a simple example of gaming between individual and collective intelligence. After my boss finishes pulling and pushing with other department heads, he often tells me: “See how this push-and-pull is reasonable and well-paced? Learn from it.” The trick I discovered in this process is comprehensible input and output. Because you can’t expect people from other departments to have the same technical background as you, and you can’t expect people from the same department to have the same technical depth as you. But in this gaming process, when your leadership tells you that you need to promote the most beneficial/suitable solution, you need to provide comprehensible output to the collective. Besides this, you can also ensure the feasibility of this process through some workflows and mechanisms—for example, using a security governance committee to avoid unreasonable decisions within the security team, using business unit weekly meetings to avoid unreasonable decisions between departments.

Note: Comprehensible input refers to acquiring i+1 (just a bit more than you understand) knowledge. It has three conditions: interesting, sufficient quantity, comprehensible. When the level of interest is greater than the stress level, more knowledge can pass through an individual’s affective filter and be effectively absorbed. This theory comes from Dr. Stephen’s English teaching. Conversely, when you provide comprehensible input to others, that’s your comprehensible output.

3. Summary

Actually, enterprises have always been relatively lenient in recruiting security operations engineers, because in most cases, you just need to quickly get started with repetitive tasks. As an architect, after tracking the SOC team’s work for a few months, I gained some insights and did some design for the SOC team, but the actual results were mediocre. Similarly, earlier on, I thought the architecture team’s output was limited. At the time, I attributed it to information gap and insufficient participation, but even after establishing a security governance committee later, it couldn’t fully solve this problem. Now I also attribute it to a failure in collective intelligence engineering. Because participation can only solve the information gap problem—whether they can understand is another matter. As the first step, you still need to solve the remaining problems through communication and “innovation.” Additionally, since the execution process is often in other teams, you need to verify execution results to a certain extent. This completes team collaboration and output.

Also, big companies doing self-development don’t need to look down on small companies, and small companies don’t need to think big companies are that impressive; those buying commercial products don’t need to think they’re much better than open source, and those using open source don’t need to envy those with budgets to procure; internet companies don’t need to boast about so-called technology, and banks don’t need to accept internet industry’s hype. There’s no best, only most suitable. Where there are advantages, there are disadvantages. High scenario-based customization means it’s not easily generalizable. Buying E5 doesn’t mean you get customization—quite a few things still require separate payment.

LLMs make the results of collective intelligence easier for everyone to access. GPT auditing contracts can dramatically improve efficiency.

References: