Defender For Cloud - under the hood: Continuous export at scale
Learn how the defender for cloud continuous export feature works
If your idea of 'continuous export' is manually copying security data into a spreadsheet at 2 AM, it might be time to let Defender for Cloud do the heavy lifting. In this blog, we’ll break down how continuous export works at scale to get all the data you need in one place.
‼️) This blog is part of a Defender For Cloud "Under the hood series" where we explore the various features, benefits and considerations that come with implementing Defender For Cloud (and other CNAPPs). You don't need any prior experience just a willingness to learn and develop. I don't quite know when the series will end but I hope you enjoy. What is continuous export?
Continuous Export in Microsoft Defender for Cloud is like setting up an automated data pipeline that keeps security insights flowing so that you can further analyse them. Instead of manually pulling reports, on your recommendations for example recommendations data can be exported to a log analytics workspace and later queried via KQL.
Continuous export ensures that security alerts, recommendations, and compliance data are continuously pushed to your Log Analytics workspace or an Event Hub, depending on what you're trying to achieve. To simplify it I've created a simple diagram for you let’s break it down.
The diagram shows how Microsoft Defender for Cloud collects security data and sends it to different locations using Continuous Export (once configured)
Data Ingestion: Defender for Cloud gathers security recommendations, alerts, and compliance data from your environment.
Continuous Export: This data is automatically sent to one or both of the following destinations:
Log Analytics Workspace: Where security logs can be analysed and used by Microsoft Sentinel for threat detection and response.
Event Hub: Which allows the data to be streamed to a third-party SIEM or other external systems for further processing.
Under the hood workbooks will also use the exported data from log analytics to present it back visually.In the console
To set up a continuous export to Log Analytics or Azure Event Hubs by using the Azure portal:
On the Defender for Cloud resource menu, select Environment settings.
Select the subscription that you want to configure data export for.
In the resource menu under Settings, select Continuous export.
We’ll stick with log analytics for the purpose of this blog:
As you can see, the export options are quite granular in terms of the type of data you can export whether it's alerts, recommendations, or severity levels. What you can select will depend on what you have enabled. For example, I can only export compliance standards that I have previously turned on.
Continuous export allows you to define how frequently data is exported. Streaming updates export data in real time, while snapshot exports provide a weekly snapshot of the selected data types under ‘Exported data types’. The supported data types include security recommendations, security findings, overall secure score, secure score controls, regulatory compliance, and security attack paths.
To configure continuous export, you’ll need to select a subscription, a resource group where your Log Analytics workspace resides, and the Log Analytics workspace itself.
Limitation of the portal
Configuring continuous export via the portal becomes impractical if you have hundreds of subscriptions. Manually going into each subscription to configure export settings is repetitive and inefficient.
Options to Scale Continuous export
Fortunately, Microsoft provides multiple ways to scale the configuration of continuous export for your Defender for Cloud data:
Continuous export in Defender for Cloud by using the REST API.
Continuous export in Defender for Cloud by using the Azure Policy covered below.
Azure Policy
To deploy continuous export configurations across your organisation, you can use DeployIfNotExist Azure Policies to automate the setup. These policies allow you to configure continuous export at scale using both Log Analytics and Event Hub.
Continuous export to Event Hubs using azure policy: Deploy export to Event Hubs for Microsoft Defender for Cloud alerts and recommendations
Continuous export to Log Analytics workspace using Azure Policy: Deploy export to Log Analytics workspace for Microsoft Defender for Cloud
I’ll select the continuous export policy for Log Analytics and proceed with configuring continuous export at scale. Once inside, I can see the generic Azure Policy without my required custom parameters. To continue, I’ll click on "Assign Policy"
Policy Basics
Next, we’ll provide the necessary custom parameters in the basics tab. It’s important to select the correct scope when configuring at scale use a management group rather than a subscription, ensuring that all child resources inherit the policy. Additionally, and optionally select the latest policy version and enable auto uplift for minor updates under "Versions". Set policy enforcement to "Enforced".
Policy Parameters
This is where we specify the parameters needed to enable continuous export at scale. I have intentionally unticked the "Only show parameters that need input or review" box to display the full underlying configuration.
Under the hood setting break down: Resource group name: The name of the resource group where the export to the Log Analytics workspace will be configured. If the specified resource group does not exist, it will be created within the subscription.
Location: The region where your current or new resource group is or will be created.
Create resource group: If a resource group does not exist within the scope, a new one will be created. If the resource group already exists and this flag is set to "true", the policy will re-deploy the resource group.
Exported data types (e.g. recommendations, alerts, attack paths): To export a snapshot (preview) of the data once a week, select the desired data types. By default, all data types are selected.
Recommendation IDs: Applicable only when exporting security recommendations. To export all recommendations, leave this field empty. To export specific recommendations, enter a list of recommendation IDs separated by semicolons (';'). Recommendation IDs can be retrieved via the Assessments API or Azure Resource Graph Explorer by selecting security resources and microsoft.security/assessments.
Recommendation severities: Choose which severity levels of security recommendations to export (e.g. Low, Medium, High).
Include security findings: Security findings are results from vulnerability assessment solutions. These findings are considered ‘sub’ recommendations grouped under a ‘parent’ recommendation.
Secure score control IDs: Applicable only for exporting secure score controls. To export all secure score controls, leave this field empty. To export specific controls, enter a list of secure score control IDs separated by semicolons (';'). These IDs can be retrieved via the Secure Score Controls API or Azure Resource Graph Explorer by selecting security resources and microsoft.security/securescores/securescorecontrols.
Alert severities: Applicable only for exporting security alerts. Determines which alert severities are exported (e.g. High;Medium;Low).
Regulatory compliance standard names: Applicable only for exporting regulatory compliance data. To export all compliance standards, leave this field empty. To export specific standards, enter their names separated by semicolons (';'). Compliance standard names can be retrieved via the Regulatory Compliance Standards API or Azure Resource Graph Explorer by selecting security resources and microsoft.security/regulatorycompliancestandards.
Log Analytics workspace: The Log Analytics workspace where the exported data will be sent.
Remediation
The Remediation tab allows you to create a managed identity (user or system-based) to automatically enforce the policy and ensure that all subscriptions are enrolled in continuous export.
I have created a remediation task so that if any subscriptions under the selected scope in my case, a management group do not have continuous export enabled to the specified Log Analytics workspace within the designated resource group, the policy and its associated managed identity will automatically enforce it.
It's important to note that you can run a remediation task at any time.
None compliance message:
You can add a non-compliance message to provide context when a resource is not meeting the policy requirements.
Azure Policy non-compliance messages appear in the Azure portal and help users understand why a policy has been flagged as non-compliant. These messages can be customised to give specific guidance on what actions need to be taken to resolve the issue.
Once deployed the policy will take 15-20 mins to take effect.Compliance state:
The compliance state of the policy allows you to monitor whether your continuous export configuration has been successfully applied across your environment.
Once the policy is deployed and the remediation task has been run, you can check its compliance status in the Azure portal under Azure Policy > Remediation tasks. This will show whether all targeted subscriptions have been correctly configured for continuous export.
The compliance state will indicate:
Compliant – The policy has been successfully applied, and all targeted resources meet the requirements.
Non-Compliant – Some resources do not adhere to the policy, meaning manual intervention or a remediation task may be needed.
Regularly reviewing the compliance state ensures that your Defender for Cloud continuous export remains correctly configured at scale. If any subscriptions are marked as non-compliant, you can trigger a remediation task to enforce the policy automatically.
Cost implications:
Saving data to a Log Analytics workspace incurs ingestion charges, which depend on the volume of data being exported. You can find detailed pricing information here and use the azure pricing calculator to estimate costs based on your specific usage.
Azure monitor:
The role of Azure Monitor in continuous export is covered in Defender for Cloud "Under the Hood": Continuous Export at Scale – Part 2.
Reference:
Learn about KQL - A book by Rod Trent Must Learn KQL
Defender For Cloud GitHub Labs
Defender for Cloud Feature requests
Defender for Cloud in the field show
Defender For Cloud Ninja training
Have blog ideas, want to engage on a topic, or explore collaboration? Let’s take it offline reach out on LinkedIn. I’d love to connect and continue the conversation!