Use Case Testing
Preparation
The first step is to analyze which types of logs exist and which could be used to detect the suspicious behavior. How is the behavior logged? Are there different systems on which logging is performed differently? Can our customers' systems generate the relevant log data or does this require additional software? Does additional log volume arise?
In the second step, we analyze how the behavior can be mapped with existing data models; this can later help us to correlate different events. The data models not only help us with the analysis, but also give us the opportunity to search efficiently in the logs and thus keep the demands on the infrastructure and therefore the costs for our customers low.
The final question is whether the suspicious behavior can be simulated. If this is the case, we are planning to integrate a procedure with which the use case can be tested end-to-end on a daily basis
Implementation
Now it's time for implementation! This begins with the normalization of the raw log data. The required field contents are extracted and assigned to the corresponding fields of the data model. For example, UserName='Admin' becomes the normalized field user='Admin'.
As soon as the data is available in the desired format, we start writing the "search". Depending on the use case, this can be very simple (for example, searching for certain field contents) or very complicated (if information from different data models needs to be merged and correlated). We note that each use case contains certain mandatory elements:
- Security domain (network, endpoint, etc.)
- Timestamps
- Possibility to suppress irrelevant incidents
- Drilldowns for analyzing the raw log lines
- Linking with existing dashboards
Whenever possible, we write a test procedure and integrate it into our end-to-end testing framework. This simulates the processes to be detected on a dedicated host on a daily basis and gives us the opportunity to determine whether all use cases are working properly.
As part of our release process, the new use cases are extensively tested and then rolled out to our customers.
Sources of error during operation
Due to the large number of systems and network architectures, the list of possible sources of error is also long. Without end-to-end testing, it is very difficult to estimate the degree to which a SIEM is available, and defects can sometimes remain undetected for years.
A selection of possible sources of error illustrates how important ongoing monitoring is:
- Change to the log format: after a software update, events are logged in a different format. A small difference can lead to individual field contents no longer being extracted correctly.
- New log source: After switching to new software (e.g. new antivirus solution), logs are delivered in a new format that the SIEM solution does not yet understand and must be normalized.
- Log forwarding is not working properly: The log volume could be too large, or certain systems deliver logs late. This can lead to logs not being searched and correlated at the right time.
- Limits: We have often observed that the number of search results is limited due to internal limits in SIEM Splunk.
- Fields without values: When aggregating by individual fields, it can happen that events containing empty fields are excluded from the search.
- Add-ons available from the manufacturer often normalize incorrectly.
- One of the processes between log generation and alerting is frozen.
Conclusion
The development of a use case is a complex matter, even for simple cases. Only in combination with an end-to-end testing framework can the reliable functioning of attack detection be guaranteed.
Supplementary information
a) A hybrid environment, in which the raw log data is first filtered in a separate central log management system, then normalized and sent to the SIEM solution.
b) And a pure Splunk environment, in which the raw logs are normalized in Splunk apps.
Glossary
Use Case
A use case defines an attack. When we talk about a use case in the context of a SIEM solution, we mean the search for suspicious behavior that is to be detected by a use case. Combined with the test procedure, see also Use Case Testing, this ensures that the search and alerting function reliably.
Emergency Use Case
We now also have the option of rolling out use cases in an accelerated process. This is useful in the event of an acute threat, for example. Later, emergency use cases are either rolled out as normal use cases or removed if the threat situation no longer exists.
Customer use case
Many of our customers have their own specifications or wishes as to what should be monitored in addition to our existing use cases. In this case, we provide advice and implement customer-specific use cases.
Operational use case
This checks the integrity of the SIEM platform. For example, we check whether log data arrives late or whether internal limits are reached.
