Audit Logging Overview
What is Audit Logging?
Audit logging is the process of documenting activity within the software systems used across your organization. Audit logs record the occurrence of an event, the time at which it occurred, the responsible user or service, and the impacted entity. All of the devices in your network, your cloud services, and your applications emit logs that may be used for auditing purposes.
What is an Audit Trail?
A series of audit logs is called an audit trail because it shows a sequential record of all the activity on a specific system. By reviewing audit logs and correlated audit trails, systems administrators can track user activity, and security teams can investigate breaches and ensure compliance with regulatory requirements.
What do Audit Logs and Audit Trails Document?
Audit logs capture the following types of information:
Event name as identified in the system
Easy-to-understand description of the event
Event timestamp
Actor or service that created, edited, or deleted the event (user ID or API ID)
Application, device, system, or object that was impacted (IP address, device ID, etc.)
Source from where the actor or service originated (country, host name, IP address, device ID, etc.)
Custom tags specified by the user, such as severity level of the event
While audit logs can take the form of a physical file, the term usually refers to digital records that you can store in a log management platform.
Audit Logs vs. Regular System Logs
The difference between audit logs and regular system logs (e.g., error logs, operational logs, etc.) is the information they contain, their purpose, and their immutability. Whereas regular system logs are designed to help developers troubleshoot errors, audit logs help organizations document a historical record of activity for compliance purposes and other business policy enforcement. A log from any network device, application, host, or operating system can be classified as an audit log if it contains the information mentioned above and is used for auditing purposes. Compliance frameworks also generally require organizations to meet long-term retention policies, which is why audit logs aim to be immutable so that no user or service can alter audit trails.
What Types of Activity Do Audit Logs Track?
Organizations typically use audit logs to track the following types of activity:
- Administrative activity
This includes events like creating or deleting a user account, such as deleting a user from your CRM tool (e.g., Salesforce).
- Data access and modification
This includes events where a user views, creates, or modifies data, such as downloading a file from payroll software (e.g., Workday).
- User denials or login failures
Audit logs such as Okta and VPN logs may capture when a user is unable to login to a system (e.g., due to invalid credentials) or is denied access to resources like a specific URL.
- System-wide changes
Audit logs from sources like AWS Cloudtrail may capture larger events occurring within a network, such as a user creating a new VM instance or creating a new application.
The decision of exactly which activity to audit is left to each organization. Systems administrators, security engineers, and human resources (HR) personnel may all wish to audit different systems for different reasons.
How Audit Logging Works
Most technologies in your tech stack will offer a UI where you can enable audit log collection. Depending on the specific tool, you may also have more granular control over audit log collection. For example, cloud vendors such as Amazon Web Services, Microsoft Azure, and Google Cloud automatically collect a wide range of audit logs. However, you may have to enable audit logging for certain services or certain types of activity to ensure you have enough data to prove compliance or investigate an incident.
Teams can send their audit logs to a central log management platform for easy storage, search, and analysis. In order to send your logs to a log management platform, you’ll need to install an agent on your hosts or utilize a direct integration between the logging platform and the software that you’re collecting audit logs from.
Benefits of Audit Logging
Whereas in the past audit logging was more common in specific industries like finance and insurance, it is now front and center for all types of companies with a digital footprint. Across industries, audit logging can be used to achieve the following important goals:
- Ensuring compliance with industry regulations
Regulations like CIS, PCI DSS, and SOC 2 affect a wide variety of industries. Audit logs can be used to show that your organization met certain benchmarks (e.g., password security for CIS) during a specific time period.
- Troubleshooting system issues
Audit logs contain detailed historical information that can be used to reconstruct the timeline of a system outage or incident. For instance, logs can help distinguish between operator error and system error. Audit trails can also be used to remediate a problem, such as potentially restoring a corrupted file to its original state by examining what changes were made to it.
- Reconstructing security breaches
When breaches occur, an audit trail can help organizations find out how they happened. For example, if an employee complains that their bank account information is incorrect in the payroll system, HR staff can examine audit logs to determine who changed the account information and when.
- Recommending new security and audit procedures
Organizations can enforce individual accountability and reduce the likelihood of security breaches or fraudulent activity by reviewing audit logs and recommending new security procedures.
- Providing legal evidence
In legal proceedings, audit logs can provide proof of validity of a specific event, such as an individual’s e-signature on a document.
Challenges of Audit Logging
The primary challenge of audit logging is knowing what to audit. In modern distributed environments, organizations often emit terabytes of log data per day. Due to cost concerns and time constraints, many organizations don’t audit all of their system activity. Companies may collect logs only from some systems or may collect only those logs that document certain types of activity (e.g., administrative activity logs but not data access logs). Alternatively, organizations may retain logs for a short amount of time to reduce storage costs. Cutting corners on which logs to collect and how often can create blind spots in your environment. As a result, you may be at risk of noncompliance or lack sufficient information for an audit or investigation. As we’ll discuss more below, some modern monitoring platforms allow you to cost-effectively ingest and archive logs for long periods of time.
Another challenge of audit logging is preventing unauthorized access to, or modification of, the audit logs themselves. The integrity of an audit log is essential to its value. An intruder may damage that integrity by modifying an audit log when attempting to cover their tracks. To prevent modification, you can restrict access to audit logs to a small number of people within your organization, or utilize other approaches for tamper-proof logging. Audit logs may also contain sensitive information such as bank account numbers. Log management tools can be used to encrypt such data to avoid unauthorized access.
Complex distributed environments also pose a challenge for audit logging. Sometimes, an intruder’s modification of one system may affect another system. For example, if a user modifies their job title in a personnel system, that may automatically trigger a salary change in the payroll system. Audit logs may capture both modifications, but someone reviewing the logs manually days or weeks after the events may find it difficult to correlate the logs together.
When searching for an audit logging solution, these are the main factors to look for:
- Completeness
The auditing solution should collect all relevant details to maintain a complete audit trail. For example, a tool that captures user activity but not location and time is incomplete.
- Consistency
To avoid using multiple different tools, an auditing solution should capture details consistently across devices and browsers. For example, a tool that captures correct historical details on web properties but not on mobile devices would be inconsistent.
- Easy parsing and querying
To efficiently analyze audit logs, the logging tool must be able to parse raw log data into structured data that contains the relevant information (e.g., event name, event description, user ID, etc.). Once parsed, an audit logging tool should also make it easy to search for specific audit logs using tags.
- Access control
Teams should have the ability to control who can view the audit logs and encrypt any sensitive data within them.
- Alerting
The relevant teams should be notified as soon as a critical event is identified by an audit log.
- Cost
An audit logging tool should provide a cost-effective way to store logs for long time periods as required by company policy or regulatory requirements.
With Datadog Log Management, you can centralize audit logs from all of your third party systems by installing the Datadog Agent or by utilizing our700+ out-of-the-box integrations. Datadog retains logs for 15 months or more, which allows for easy historical analysis and investigation. You can use filters like user email, API Key ID, or Method (e.g. Post, Get, Delete) to search through audit logs at any scale and retrieve those most essential for review. Finally, Datadog’s Cloud SIEM analyzes ingested audit logs for threats in real time, helping you detect and stay ahead of security incidents.
To track activity within Datadog itself, you can leverage the Audit Logs Explorer, which records all calls made to Datadog’s API and product-specific changes. Having this audit trail makes it easy to, for example, view all changes leading up to a breaking event or determine if someone modified a log processing pipeline, causing a dashboard or monitor to break.
