Best Practice for Incident Forensics

Published on 01 April 2015

Today security incidents are a question of when, not if: every company, small and large, faces the threat of being a cyber-victim. There are numerous ways hackers can cause serious loss and damage to a company. If the crack happened you cannot do anything but rely on your logs generated by network devices and applications. Those are critical to determining the initial cause. Still, many companies face significant challenges during forensic investigations.

My blog was published as an opinion article at

Challenges to overcome

Binary Code AbstractFirstly and most importantly, it is difficult to access the logs. They might reside in different locations or on different systems and, therefore, getting to the bottom of an incident becomes more difficult and takes more time.

If you don’t have all the pieces and information that are required, it might happen that you miss that exact piece that serves as your evidence. Without having that, piecing together what happened in your system makes the whole investigation more time-consuming and reduces confidence in the investigation conclusions.

If you gathered all the information required that must be an overwhelming amount of data. The sheer amount of log data to sift through in a forensic investigation can delay detection and resolution. Searching on extremely large amounts of data can take days when you want the answer in seconds.

Logs provided in an unstructured way which makes your investigation even slower. Many companies struggle to make sense of log data that has varying formats and structures, sometimes for the same type of event.

Poor data integrity also challenges the investigation. Once you’ve found out what happened, you need logs that meet the legal standard for evidence. Logs that have been transformed from their original format or have not been securely stored may not be accepted as evidence in a court of law.

Solution for easier investigation

As you can see above, not only the fact that you have been hacked means the problem to your company, there is still a lot to do even afterwards. And once you had the trouble occurred you don’t want to spend time needlessly. In crisis what you really need is to be as quick as possible: you want to search the logs quickly and efficiently, you want a reliable log transfer, distributed pre-procession, and tamper-proof storage. Let’s see how you can solve these problems.

An indexing engine and user interface will help you to search terabytes of data quickly. Syslog-ng Store Box does have it. You can rely on syslog-ng log transfer as they can both ensure zero message loss during transport from clients to the central log server.

Furthermore, both products use SSL/TLS encryption to transfer logs and the log store, an encrypted, time-stamped and digitally signed log file. Above all these, syslog-ng can filter, parse, re-write and classify data on clients at unparalleled speeds to reduce the size and complexity of log data stored centrally, therefore, providing distributed pre-procession.


You may wonder what kind of benefits it provides you. The fact that you are able to segment and search mountains of log data allows you for faster root cause analysis and remediation. The tamper proof, secure logs in their raw format provide legally admissible evidence so you have higher quality data in your hands. And last but not least, your improved confidence in your investigation will help your whole work: being certain that logs aren’t missing or haven’t been tampered with increases the confidence in the results of your investigation.


Is the problem solved?forensics

Let’s assume you have gathered all the logs, you’ve made them indexed in order to make them easier to search. You search them but still don’t find that step that is missing from the whole story. System management tools are improving companies’ ability to handle system error, but the solution to human error or targeted attacks still remains elusive. Cyber attackers increasingly hijack administrative accounts gaining privileged access to the whole IT environment without strict control. Without the reliable recording of administrative access to the servers, the investigation of incidents becomes expensive and circumstantial.

In addition, external standards such as the ISO 2700x or the PCI-DSS specify strict measures to support future investigations, by requiring the recording of user activities or fault logging in place. Without user session recording the question of who did what is almost impossible to answer, and often leads to accusations along with the time and money wasted on investigating the incident. To avoid this, a tamper-proof session recording solution could be implemented.

Eliminating blind spots

Shell Control Box (SCB) is a cost-efficient and compliant solution to aid in the investigation of incidents related to IT systems. For example, in case of an unexpected shutdown, data leakage, or database manipulation, the circumstances of the event are readily available in audit trails so the cause of the incident can be quickly identified. The recorded audit trails can be played back like a movie – recreating all actions of the user. Consequently, SCB helps to find not only the root cause of a problem, but also the responsible person. This is especially important in case of business-critical systems, or if the company has outsourced its IT administration to an external company.

Audit trails are invaluable for both real-time and post-mortem investigations. They enable the internal auditor to search, for example, for all the users who accessed a specific account number in a specific time-frame across any platform in the enterprise. As audit trail content can be easily interpreted, SCB eliminates the need for costly external consultants in the case of forensics investigations. SCB prevents anyone from modifying the audited information as audit trails are time stamped, encrypted and signed. This makes SCB capable of reconstructing events and providing tamper-proof evidence in case of legal proceedings, too.

Lowering forensics and troubleshooting costs

SCB is an independent network device that operates transparently, and extracts audit information directly from the communication between the client and the server. Audit trails can be browsed online or viewed real-time to monitor the activities of the administrators. The web-based video player enables fast forwarding during replays, as well as free-text search for events making forensics investigations quick and cost-efficient. By the free-text search capability commands entered by the user or displayed texts in graphical protocols (e.g. in RDP) can also be searchable. It is also possible to execute searches on a large number of audit trails to find sessions that contain a specific information or event.


In addition to recording audit trails of the inspected protocols, embedded protocols (for example, other protocols tunneled in SSH, port-forwarding) and file transfers can be recorded as well. Transferred files from SCP and SFTP connections can be extracted for further analysis. It is even possible to convert the audited traffic into packet capture (pcap) format to analyze it with external tools. For example, if a service becomes unavailable, you can get a list of users who recently accessed the server, check the type of their access (file transfer, shell, etc.) to find which might affect the service, check the content of transferred files, get a list of commands typed or replay the suspicious sessions.

Solution in a nutshell

The simple question “Who accessed our server and what did he do?” is one of the toughest questions to answer in IT today. Companies might be hacked, hit with denial of service, be a victim of fraud, or their sensitive data might be stolen. You can’t do anything against it but can help the investigation with previous efforts: collect and store your data structured and make sure you don’t have lost messages. Always be prepared for the blind spots and be cautious with the events that are not seen in logs.

Balabit’s Contextual Security Intelligence solution protects organizations in real-time from threats posed by the misuse of high risk accounts. The solution includes reliable Log Management providing context aware data ingestion, Privileged User Monitoring and User Behavior Analytics to monitor user activities and gather audit data and events from multiple sources. Working in conjunction with existing control-based strategies Balabit enables a flexible and people-centric approach to improve security and compliance without adding additional barriers to business practices.

Learn more on syslog-ng and SCB.

by Gabor Marosvar

share this article
Mitigate against privileged account risks
Get in touch

Recent Resources

Audit Reporting in Shell Control Box

Shell Control Box (SCB), Balabit' privileged u er monitoring appliance, feature  a comprehen ive reporting y tem, which...

Best Practice for Incident Forensics

Today ecurity incident are a que tion of when, not if: every company, mall and large, face the threat of being a...

Calibrating anomaly scores

In thi blog po t, we are going to di cu how to calibrate anomaly core to make ure that the core are tru ted...

“Lorem ipsum dolor sit amet, consectetur adipisicing elit. Sint minima earum velit, dolorem fuga impedit onsectetur adipisicing dolorem.”

– Lorem ipsum, Lorem ipsum