Achieving Strong Coordination Between ITOps and SecOps via a Unified AIOps Platform
December 6, 2023
Tracing the Origins of GoogleUpdater: Uncovering the Provenance Chain
February 22, 2024
Introduction
Modern operating systems have rich logging capabilities, useful for detecting a wide range of security threats. While the Federal Government mandates standards for configuring these logging systems, the standards are weak and unable to detect anything beyond the most straightforward misbehaviors. The problem stems from the fact that the government must define a one-size-fits-all static logging configuration that works across a wide range of organizations whose computing requirements (e.g., locations such as cloud, on-premise, data center, etc., or form factors, such as physical/virtual machines, containers, etc.) can vary dramatically over time.
Let us present some history on endpoint logging from the context of 30+ years of intrusion detection and analysis performed by Ennetix team members and others. There is a thread that runs from the 1987 National Telecommunications and Information Systems Security Committee directive NTISSP No. 200 and the “Computer Security Act of 1987” to the 2021 Presidential Executive Order 14028 “Improving the Nation’s Cybersecurity”. Federal government organizations and Federal contractors were directed to bring their computer systems’ security up to the NSA’s C2 level of computer security. In the security world, this was referred to as “C2 by ‘92”.
NSA’s C2 standard required that the Operating Systems (OSs) used by computing equipment in the government and by government contractors to generate detailed audit logs of activity occurring on the computer, including the creation of processes, files, network connections, etc. These logs could be used to detect attacks and reconstruct attacks. By 1992, several computer system vendors met the requirement, including Sun Microsystems, which developed the Basic Security Module (BSM) that added the audit requirements. Over time, several other OS platforms adopted the BSM auditing subsystem, including OpenBSD and Apple’s Mac OS. By 2004, virtually every major OS vendor had a rich audit system that could generate detailed audit trails of activity on their system. As OS platforms added these rich telemetries to their kernels, Intrusion Detection System (IDS) developers were demonstrating that this logging data, when fed into IDSs, was very effective in detecting a wide range of threats, including unknown attacks against unknown vulnerabilities.
So, for over 25 years, if we have had logging and associated IDSs that were very effective at detecting attacks, why are so many attacks occurring and going unnoticed today? The answer is that logging systems are largely disabled. Government compliance organizations such as the Defense Information Systems Agency (DISA), National Institute of Standards and Technology (NIST), and Committee on National Security Systems Instruction (CNSSI), and non-government compliance entities such as the Center for Internet Security (CIS) direct organizations to turn off logging of key activities such as: (a) executing a program, (b) initiating or accepting a network connection, and (c) opening a file. Result: the following attack could be carried out against these systems with zero evidence of the attack recorded by audit systems:
“A malicious program is started. It reads all files owned by the user and bundles them up in a tarball or zip file. It initiates a connection to a remote host and exfiltrates all the user’s data over that connection. It then makes encrypted versions of the files (for ransomware purposes) and deletes the original files, leaving the user without access to their files. The program then deletes itself and exits.”
The audit systems, as configured by these organizations, generate no data that an IDS can use to detect the attack. There are no audit logs that forensic examiners can use to reconstruct the attack. Why do standards bodies such as DISA, NIST, and CNSS direct organizations to turn off useful logging? In over 25 years of asking this question, there were no official answers. Unofficially, the perception is the issue is performance. Turning on auditing to a level that is effective for IDS and forensics consumed too many resources – CPU, storage, and network bandwidth. A critical lesson is that effective security that impacts performance will be disabled.
Alternatives to Compliance-Mandated Endpoint Logging
Below we discuss two popular alternatives to host-based IDS using native OS logging data.
A. Network-based Intrusion Detection Systems (NIDS): While compliance organizations were directing people to disable endpoint logging, network monitoring, and forensics tools took off in the mid-1990s, such as Air Force’s ASIM global sensor grid, DISA’s Joint Intrusion Detections System (JIDS), DOE’s Network Intruder Detector (NID) – which were all based on the UC Davis Network Security Monitor (NSM) that was developed by Ennetix researchers – Snort, Bro (renamed Zeek), and Ethereal (renamed Wireshark). Snort, Zeek, and Wireshark are all still actively used today. The Cybersecurity and Infrastructure Security Agency (CISA) deploys the EINSTEIN network monitoring platform to protect the Federal Civilian Executive Branch (FCEB). For any commercial IDS platform, a network-based IDS (NIDS) is table stakes. However, NIDS solutions face several challenges that are becoming more problematic over time:
- Encryption: When NIDS became the dominant form of IDS in the mid-to-late 1990s, very little network traffic was encrypted. This allowed sensors to leverage Deep-Packet Inspection (DPI) for both detection and forensics (e.g., generating transcripts of keystrokes for Telnet logins, commands in FTP sessions, etc.) Today, encryption is ubiquitous, limiting the effectiveness of NIDS for detection and forensics.
- Network Address Translation (NAT): In the mid-to-late 1990s, each device had a publicly-routable IP address. As the pool of IPv4 addresses began to run out, these devices were increasingly placed on private address blocks and connected to the Internet via NAT routers, as one single IP address represented many devices behind the NAT router. Within an enterprise, there are often layers of NAT routers. If a NIDS is placed outside a NAT router, there are at least two issues. First, the behavior of a single misbehaving device can be masked by the behaviors of many non-compromised devices, i.e., the signal-to-noise ratio is decreased by aggregating many devices to a single IP address (“signal” is an observable activity created by a compromised device and “noise” is all activity created by all the non-compromised devices). Second, even if malicious behavior is detected, identifying the compromised device behind the NAT router can be challenging.
- Cloud: Increasingly, organizations are hosting data in cloud services such as AWS. Both good and bad actors leverage these cloud services to move data to and from these services. With all parties (good and bad) using the same network resources, traffic analysis (analyzing which devices communicate with which other devices) becomes much less effective. The remote organization (e.g., AWS, Azure, Hurricane Electric, Digital Ocean, etc.) tells us little about whether the connection is legitimate or malicious.
B. HIDS, EDR, XDR: The community appears to increasingly recognize the limitations of NIDS, so endpoint solutions are becoming increasingly popular. On May 12, 2021, President Biden signed Executive Order 14028, “Improving the Nation’s Cybersecurity,” which called out the need to widely deploy Endpoint Detection and Response (EDR) solutions. This was followed by the October 8, 2021, Office of Management and Budget (OMB) memorandum M-22-01 with the subject: “Improving Detection of Cybersecurity Vulnerabilities and Incidents on Federal Government Systems through Endpoint Detection and Response.” M-22-01 provides “implementation guidance to agencies to accelerate the adoption of EDR solutions.”
Many solutions that self-identify as EDRs exist. A search for EDR, eXtended Detection and Response (XDR), or Host-based Intrusion Detection System (HIDS) shows many such systems. But customers typically have no idea what is under the hood of any particular EDR solution. Compare this to the logging architectures developed and evaluated over many years to meet the requirements of compliance bodies such as DISA and NIST. For example, DISA’s Security Technical Implementation Guide (STIG) for Red Hat Enterprise Linux 8 includes 82 specific requirements for its logging system, covering topics ranging from ownership and permissions for files and programs to what specific activities should be recorded in the logs.
Recent Federal efforts such as the 2021 Executive Order 14028 and OMB memorandum M-22-01 share similar goals just like the 1987 National Telecommunications and Information Systems Security Committee (NTISSC) directive NTISSP No. 200 and the Computer Security Act of 1987. They focus on collecting relevant information to detect threats, hunt for unknown threats, and provide forensic evidence on end hosts.
Conclusion
NIDS, despite the above challenges, will continue to be a vital part of cybersecurity detection and response, but the larger community, including the executive branch, recognizes the critical need for EDR capability. The next generation of security should consider end-to-end coverage through both NIDS and endpoint logging.
