What's New in Microsoft Sentinel and XDR: AI Automation, Data Lake Innovation, and Unified SecOps

I originally published this article on the Microsoft Tech Community.

The most consequential “new” Microsoft Sentinel / Defender XDR narrative is the operational and engineering shift to unified security operations in the Microsoft Defender portal, including an explicit Azure portal retirement/sunset timeline and concrete migration implications (data tiering, correlation-engine changes, schema differences, automation behavior changes). Official sources now align on March 31, 2027 as the sunset date for managing Microsoft Sentinel in the Azure portal, with customers redirected to the Defender portal after that date.

The headline feature announcements to anchor around — because they create new engineering patterns, not just UI changes — are:

• AI playbook generator (preview): natural-language authoring of Python playbooks in an embedded VS Code environment (Cline), using Integration Profiles for dynamic API calls and an Enhanced Alert Trigger for broader automation triggering across Sentinel, Defender, and XDR alert sources.
• CCF Push (public preview): a push-based connector model built on the Azure Monitor Logs Ingestion API, where deploying via Content Hub can auto-provision the typical plumbing (DCR/DCE/app registration/RBAC), enabling near-real-time ingestion plus ingestion-time transformations and direct delivery into certain system tables.
• Data lake tier ingestion for Advanced Hunting tables (GA): direct ingestion of specific Microsoft XDR Advanced Hunting tables into the Sentinel data lake without requiring analytics-tier ingestion — for long-retention, cost-effective storage and retrospective investigations at scale.
• Microsoft 365 Copilot data connector (public preview): ingests Copilot audit/activity events via the Purview Unified Audit Log into a dedicated CopilotActivity table, with admin-role requirements and cost notes.
• Multi-tenant content distribution expansion: distributes analytics rules, automation rules, workbooks, and built-in alert tuning rules across tenants via distribution profiles (with limitations — e.g. automation rules that trigger a playbook can’t currently be distributed).
• Alert schema differences for “standalone vs XDR connector”: a must-cite engineering artifact documenting breaking/behavioral differences (CompromisedEntity semantics, field mapping changes, alert filtering) when moving to the consolidated Defender XDR connector path.

What’s new and when

Officially documented Sentinel and Defender XDR features relevant to a “new announcements” article. Where a source doesn’t explicitly state GA/preview or a date, it’s marked unspecified.

Technical architecture and integrations

Microsoft’s integration docs describe two “centers of gravity”: in Defender portal mode, Sentinel data is ingested alongside organizational data into the Defender portal for a unified surface; in Azure portal mode, Defender XDR incidents/alerts flow via Sentinel connectors and analysts work across both experiences.

On integration behavior: supported Defender components (Endpoint, Identity, Office 365, Cloud Apps), plus Purview DLP and Entra ID Protection, surface alerts through the integration. When onboarding Sentinel to the Defender portal with Defender XDR licensing, the Defender XDR connector is set up automatically and component alert-provider connectors are disconnected. Defender XDR incidents typically appear in Sentinel within ~5 minutes. Defender XDR alerts and incidents that populate SecurityAlert/SecurityIncident are synchronized at no charge, while other data types (e.g. Advanced Hunting tables) are charged.

Telemetry, schemas, analytics, automation, and APIs

CCF Push and the push-connector path

CCF Push frames the old model as predominantly polling-based and introduces push connectors where partners/customers send data directly to a workspace, with “Deploy” auto-provisioning DCE, DCR, Entra app registration + secrets, and RBAC. It’s built on the Logs Ingestion API: a sender app authenticates via an app registration with access to a DCR, sends JSON matching the DCR’s structure to a DCR endpoint or DCE (DCE required for Private Link), and the DCR can apply a transformation to map/filter/enrich before writing to the target table.

DCR transformation (KQL)

// Keep only Critical events
source | where severity == "Critical"

// Drop a noisy/unneeded column
source | project-away RawData

// Enrich with a simple internal/external IP classification (example)
source | extend IpLocation = iff(split(ClientIp,".")[0] in ("10","192"), "Internal", "External")

A Sentinel-specific nuance: Sentinel-enabled Log Analytics workspaces aren’t subject to Azure Monitor’s filtering ingestion charge, regardless of how much a transformation filters (other Azure Monitor transformation cost rules still apply in general).

Key tables to call out

• Copilot connector → CopilotActivity (record types like CopilotInteraction); enabling requires Global Administrator or Security Administrator.
• Defender XDR incident/alert sync → SecurityAlert and SecurityIncident (no charge); other Defender data types (e.g. DeviceInfo, EmailEvents) are charged.
• UEBA core tables: IdentityInfo, BehaviorAnalytics, UserPeerAnalytics, Anomalies.
• UEBA behaviors layer tables: SentinelBehaviorInfo, SentinelBehaviorEntities (only if the behaviors layer is enabled).
• XDR Advanced Hunting lake-tier ingestion (GA): supported tables include DeviceProcessEvents, DeviceNetworkEvents, EmailEvents, UrlClickEvents, CloudAppEvents (MDI support to follow).

UEBA and graph

UEBA uses machine learning to build behavioral profiles and detect anomalies versus baselines, with peer-group analysis and blast-radius evaluation. Two scoring models exist: BehaviorAnalytics.InvestigationPriority (0–10) vs Anomalies.AnomalyScore (0–1), with different processing characteristics (near-real-time/event-level vs batch/behavior-level).

The Sentinel data lake frames a two-tier model: an analytics tier for high-performance real-time analytics (alerting/incident management) and a data lake tier for centralized long-term storage and Python-based analytics, designed for retention up to 12 years with single-copy mirroring. If you already have the data lake, the required graph is auto-provisioned on sign-in to the Defender portal, enabling hunting graph and blast radius.

AI playbook generator: constraints that matter

• Prerequisites: Security Copilot enabled with SCUs available (not billed for generation but required); workspace onboarded to Defender.
• Roles: Sentinel Contributor for authoring Automation Rules; a Detection tuning role in Entra to use the generator (permissions may take up to two hours).
• Integration Profiles: Base URL + auth method + credentials; URL/auth can’t change after creation; supports OAuth2 client credentials, API key, AWS auth, Bearer/JWT, etc.
• Limitations: Python only, alerts as the sole input, no external libraries, max 100 playbooks/tenant, 10-minute runtime, line limits, and no automatic migration between enhanced-trigger and standard-alert-trigger rules.

APIs and CLI

Create/update a DCR with Azure CLI:

az monitor data-collection rule create \
  --location 'eastus' \
  --resource-group 'my-resource-group' \
  --name 'my-dcr' \
  --rule-file 'C:\MyNewDCR.json' \
  --description 'This is my new DCR'

Send logs via the Azure Monitor Ingestion client (Python):

import os
from azure.identity import DefaultAzureCredential
from azure.monitor.ingestion import LogsIngestionClient

endpoint = os.environ["DATA_COLLECTION_ENDPOINT"]
rule_id = os.environ["LOGS_DCR_RULE_ID"]           # DCR immutable ID
stream_name = os.environ["LOGS_DCR_STREAM_NAME"]   # stream name in DCR

credential = DefaultAzureCredential()
client = LogsIngestionClient(endpoint=endpoint, credential=credential)

body = [
    {"Time": "2026-03-18T00:00:00Z", "Computer": "host1", "AdditionalContext": "example"}
]

# Actual upload method name/details depend on SDK version and sample specifics.
# Refer to official ingestion samples and README for the exact call.

Sentinel repositories API retirement: older REST API versions for Sentinel Repositories retire June 1, 2026, and Source Control actions on older versions stop being supported June 15, 2026 — critical for content-as-code pipelines.

Migration and implementation guidance

Treat migration as a set of gating checks:

• No extra cost to transition to the Defender portal (billing remains Sentinel consumption).
• Policy changes: after onboarding, Defender XDR data storage/privacy policies apply even when working with Sentinel data.
• CMK constraint: customer-managed keys aren’t supported for data in the Sentinel data lake; CMK-enabled workspaces aren’t accessible via lake experiences.
• Region/residency: the lake is provisioned in the primary workspace’s region; onboarding may require consent to ingest M365 data into that region.
• Tier-switch lag: enabling ingestion or switching tiers can take 90–120 minutes for data to appear.

Enable lake-tier ingestion for Advanced Hunting tables (GA)

1. Defender portal → Microsoft Sentinel → Configuration → Tables
2. Select a supported Advanced Hunting table
3. Data Retention Settings → choose “Data lake tier”, set retention, save

Defender data stays accessible in the Advanced Hunting table for 30 days while a copy goes to the lake for long-term retention (up to 12 years) and graph/MCP scenarios.

What breaks (grounded in Microsoft’s own lists)

• Schema/field drift: CompromisedEntity behavior differences, field mapping changes, filtering differences (e.g. Defender for Cloud informational alerts not ingested; Entra ID below High not ingested by default).
• Automation delays/unsupported actions: automation rules may run up to 10 minutes after changes due to forwarding; some playbook actions (adding/removing alerts from incidents) aren’t supported after onboarding.
• Incident sync boundaries: incidents created in Sentinel via API/Logic App/manual Azure portal aren’t synchronized to the Defender portal.
• Advanced hunting differences: auxiliary log tables are no longer available in Defender Advanced hunting once the data lake is enabled; access them via lake exploration KQL.
• CI/CD failures: repository tooling calling older API versions must migrate by June 1, 2026.

Old vs new (Microsoft)

Use cases and customer impact

• Copilot governance and misuse detection — the Copilot connector’s record types enable detections for unauthorized plugin/workspace/prompt-book operations and anomalous interactions.
• Long-retention hunting — lake-tier ingestion for Advanced Hunting tables targets scale, cost containment, and retrospective investigations while keeping 30-day availability in the AH tables.
• Faster automation authoring — the playbook generator replaces rigid templates with dynamic API calls via Integration Profiles.
• Multi-tenant SOC standardization — content distribution replicates detections, automation, and dashboards across tenants while keeping execution local.

Vendor-stated metrics (treat as marketing unless independently corroborated): Microsoft’s March 2026 roundup cites a “44% reduction in total cost of ownership” and “93% faster deployment times” for organizations using Sentinel.