Below is an article about three different methods to store logs in Azure/Microsoft Sentinel for a long term storage.
SPOILER ALERT: Sentinel Archive Tier is a better approach for most scenarios 🙂
Pre-requisites
Overview
As discussed in the previous article, Microsoft Sentinel is not a platform to dump all the logs in and use as a storage box. Although organizations would require some time to store heavy volume/low value logs in "some" storage for two reasons:
Below are the three most common/preferable methods used for storing logs in Azure environment for long term retention:
Azure Blob Storage
Azure Blob Storage "used to be" the only method to store logs for a long term retention, due to its cheaper cost. This method was preferred mainly for scenarios where the long term retention of logs is mainly for compliance requirements (and not for retro-threat hunting).
Analogy
Storing logs in Azure Blob Storage is similar to,
You rarely want to view/edit/use the files in the external hard drive. And when you want it, you will have to take some effort to migrate the files from the external hard drive to your computer - to view/edit/use it.
Pros
✅ Cheapest of all the listed methods
✅ Easy to maintain and manage
Cons
❌ Requires building an automation to move the logs from Azure Blob Storage (Cold Storage) to Log Analytics Workspace (Hot Storage)
Azure Data Explorer
Azure Data Explorer was preferred (or forced to prefer) mainly by large organizations where it was a requirement to have high volume/low value logs on a hot storage. A method to store high volume/low value logs on hot storage isn't the optimized approach - especially when its putting a hole in the wallet.
Analogy
Storing logs in Azure Data Explorer is similar to,
You rarely want to view/edit/use the files in the laptop. And when you want it, you don't have the migrate the files from the laptop to your computer. Instead you can view/edit/use the files directly in the laptop. Although you need to make sure the laptop is maintained regularly and has power in the battery.
Pros
✅ Easily accessible with no custom automation to access the logs
Cons
❌ Expensive - ingestion cost and data export cost
❌ Requires maintenance and management
Microsoft Sentinel Archive Tier
Microsoft Sentinel Archive Tier was announced around early 2022 to have the benefits of both - Blob Storage and Azure Data Explorer. This method is resolving issues for both scenarios - compliance requirements and retro-threat hunting in the most optimized approach.
Analogy
Storing logs in Microsoft Sentinel Archive Tier is similar to,
You rarely want to view/edit/use the files in your computer. And when you want it, you don't have to "migrate" the files, but just decompress the files from the folder (which doesn't involve much effort).
Pros
✅ Easily accessible with a few clicks
✅ Not as expensive as Azure Data Explorer
✅ Doesn't require any maintenance or management
Cons
❌ There's a very minimal cost for running search and restore jobs
The only time we would require running search and restore jobs on Microsoft Sentinel is when the organization hits a P1 incident - and the cost for search and restore would be considered peanuts when compared with the impact of the incident
Conclusion
Azure Blob Storage was the cheapest and only option - when Microsoft Sentinel was first released. Azure Data Explorer was the second best option (if the organization is happy to accept the cost) to keep the logs in a hot storage.
Microsoft Sentinel Archive Tier is the best of both worlds where the solution was designed to be cheaper, as well as easily accessible. Azure Blob Storage still lives as a valid solution till date - when it comes to storing low value logs "purely" for Compliance requirements only.
Microsoft Sentinel Archive Tier can be used for 80 to 90% of the scenarios, and Azure Blob Storage could resolve the remaining. 🙂
Shout out to Benjamin Kovacevic for providing insights on the above
