Tips from the Trenches: Multi-Tier Logging

Tips From the Trenches: Multi-Tier Logging

Here’s a stat to make your head spin: Gartner says that a medium-sized enterprise creates 20,000 messages of operational data in activity logs every second. That adds up to 500 million messages — more than 150 GB of data — every day. In other words, as security professionals, we all have logs. A lot of logs. So, how do we know if our log collection strategy is effectively meeting our logging requirements? Unfortunately, a one-size-fits-all logging solution doesn’t exist, so many leading security teams have adopted a multi-tier logging approach. There are three steps to implementing a multi-tier logging strategy:

“ A one-size-fits-all logging solution doesn’t exist, so many leading security teams have adopted a multi-tier logging approach. ”

1. Analyze your logging requirements

A multi-tier logging strategy starts with analyzing your logging requirements. Here’s a simple checklist that I’ve used for this:

Who requires access to the organization’s logs?

  • Which teams require access?
  • Is there unnecessary duplication of logs?
  • Can we consolidate logs and logging budgets across departments?

What logging solutions do we currently have in place?

  • What is the current health of our logging systems?
  • Are we receiving all required logs?
  • Have we included all required log source types?
    • Do we need public cloud, private cloud, hybrid cloud and/or SaaS logs?
  • How many events per second (EPS) are we receiving?
  • How much log storage (in gigabytes) are we using now?
  • What are our logs of interest?
    • Create alerts and/or reports to monitor for each.

What time zone strategy will you use for logging? 

  • How many locations are in different time zones across the organization?
  • Will you use a single time zone or multiple time zone logging strategy?

How much storage capacity will be needed for logging for the next 3-5 years?

Do we have a log baseline in place?

  • Where are our logs stored now?
  • Where should they be stored in the future?

Are we collecting logs for troubleshooting, security analysis and/or compliance?

  • What are our compliance requirements?
    • Do we have log storage redundancy requirements?
    • What are our log retention requirements?
    • Do we have log retention requirements defined in official policy?
  • What logs do we really need to keep?
    • Identify those that are useful.
    • Drop those that are not.

2. Digest log information

After all of this information is gathered, it’s time to digest it. It’s important to align your logging infrastructure to log type and retention needs — so you don’t end up inserting a large amount of unstructured data that you will need to be able to quickly search in an SQL database, for example. Most organizations have multiple clouds, many different devices that generate different log types and separate required analysis methods. In other words, one solution usually does not meet all logging needs.

3. Implement multi-tier logging

If, after analyzing your logging requirements, you find that one logging strategy does not meet all of your requirements, consider this tiered logging flow:

Code42 Tiered Logging Flow Example

In this example logging flow, there are three different logging flow types and five different log repositories. There are SIEM logs, application logs and system log flow types. The repositories are the SIEM database, ELK (elasticsearch, logstash and kibana) stack, two long-term syslog archival servers and cloud storage. The repositories each have a unique role:

  • The SIEM correlates logs with known threats.
  • The ELK stack retains approximately 30-60 days of logs for very fast searching capabilities.
  • The two syslog archival servers store the last three to seven years of syslog and application logs for historical and regulatory purposes. One syslog archival server is used for processing logs, the other is a limited-touch, master log repository.
  • Cloud storage also stores the last three to seven years of logs for historical and regulatory purposes.

Simplify your log activity

This is just one quick example of an innovative solution to simplifying log activity. Regardless of whether multi-tier logging is the right solution for your organization, the most critical step is making sure you have a clearly defined logging strategy and an accurate baseline of your current logging state. This basic analysis gives you the understanding and insights you need to simplify log activity — making it easier to accomplish the complex logging goals of your organization.

Facebook Twitter Google LinkedIn YouTube