Listen to this Post
Introduction:
For years, data engineers and AI practitioners have been forced to choose between best-of-breed tools and a unified data estate—often ending up with duplicated data, complex pipelines, and governance nightmares. The latest updates from Microsoft and Databricks finally crack this problem wide open, delivering true bi-directional interoperability between Azure Databricks and Microsoft Fabric, all powered by OneLake. This isn’t just another feature update; it’s a fundamental shift in how organizations manage their data foundation, enabling teams to use the tools they love without duplicating data or building complex pipelines.
Learning Objectives:
- Understand the new Unity Catalog and OneLake integration capabilities, including managed table storage and catalog federation
- Learn how to configure bi-directional read/write interoperability between Azure Databricks and Microsoft Fabric
- Master the step-by-step setup process for OneLake catalog federation, external locations, and the “Publish to Fabric” workflow
- Implement security best practices and governance controls when bridging these two platforms
You Should Know:
- OneLake as Native Storage for Unity Catalog Managed Tables (Beta)
The most significant announcement is that Unity Catalog managed tables can now be stored directly in OneLake, eliminating the previous reliance solely on ADLS Gen2 for UC tables. This means OneLake becomes a native storage layer for Delta tables, accessible from both Azure Databricks and Microsoft Fabric with full governance, lineage, and optimization from Unity Catalog still applied.
Step-by-Step Guide: Connect Azure Databricks to a OneLake External Location
To enable Unity Catalog managed tables in OneLake, you need to configure an external location that points to your OneLake storage:
Prerequisites:
- Databricks Runtime 18.1 or above
- Active Microsoft Fabric workspace with capacity assigned
- Unity Catalog-enabled Azure Databricks workspace
– `CREATE STORAGE CREDENTIAL` and `CREATE EXTERNAL LOCATION` privileges on the Unity Catalog metastore
Step 1: Configure Identity and Access in Azure and Fabric
– In the Azure portal, create or identify an Access Connector for Azure Databricks with a managed identity, or use a service principal
– In Microsoft Fabric, navigate to your target workspace → Workspace settings → Manage access
– Add your managed identity or service principal and assign it Administrator, Member, or Contributor role
Step 2: Provision Storage in Fabric and Get the Path
– In your Fabric workspace, create an Azure Databricks Storage item
– After creation, construct the ID-based ABFSS path:
abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/
– The workspace ID and Azure Databricks Storage item ID are available in the browser URL when the item is open in Fabric
Step 3: Create Storage Credential and External Location in Unity Catalog
Using Databricks CLI or SQL:
-- Create storage credential with managed identity CREATE STORAGE CREDENTIAL oneLakeCredential WITH MANAGED IDENTITY 'arn:azure:msi:...'; -- Create external location pointing to OneLake CREATE EXTERNAL LOCATION oneLakeLocation URL 'abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/' WITH (STORAGE CREDENTIAL oneLakeCredential);
Step 4: Create Managed Tables in OneLake
Once the external location is configured, any managed table created in Unity Catalog can be stored directly in OneLake:
CREATE TABLE myCatalog.mySchema.myTable USING DELTA LOCATION 'abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/myTable' AS SELECT FROM sourceTable;
2. Enable OneLake Catalog Federation for Read-Only Queries
OneLake catalog federation allows Unity Catalog queries to run directly against OneLake storage without copying data. This brings powerful analytics and AI/BI capabilities in Azure Databricks directly to your OneLake data.
Step-by-Step Guide: Set Up OneLake Catalog Federation
Prerequisites:
- Workspace enabled for Unity Catalog
- Databricks Runtime 18.0 or above
- SQL warehouses must use 2025.40 or above
- Fabric administrator must enable: Service principals can use Fabric APIs, Allow apps running outside of Fabric to access data via OneLake, and Use short-lived user-delegated SAS tokens
Step 1: Set Up Azure Authentication
OneLake federation supports two authentication methods:
- Azure Managed Identity via an Access Connector for Azure Databricks
- Azure service principal
Step 2: Create a Connection
-- Must be metastore admin or have CREATE CONNECTION privilege CREATE CONNECTION oneLakeFederation TYPE ONELAKE OPTIONS ( tenantId '<your-tenant-id>', workspaceId '<fabric-workspace-id>' );
Step 3: Create a Foreign Catalog
-- Must have CREATE CATALOG permission on the metastore CREATE FOREIGN CATALOG oneLakeData USING CONNECTION oneLakeFederation OPTIONS ( lakehouse '<lakehouse-1ame>' );
Step 4: Query OneLake Data Directly
-- Now you can query OneLake data as if it were native Unity Catalog tables SELECT FROM oneLakeData.<schema>. < table> LIMIT 100;
Important: Data access through catalog federation is read-only. For write access, use the external location approach described in Section 1.
3. Publish to Fabric (Preview): One-Click Catalog Sharing
The “Publish to Fabric” workflow (Preview) provides a streamlined way to surface Unity Catalog tables in Fabric straight from Databricks Catalog Explorer. This creates mirrored catalog items from Databricks so the same tables can be queried across Fabric workloads without copying data.
Step-by-Step Guide: Publish a Unity Catalog Catalog to Microsoft Fabric
Step 1: Prepare Your Environment
- Ensure your Azure Databricks workspace is Unity Catalog-enabled
- Verify you have the necessary permissions as a catalog admin
Step 2: Publish from Catalog Explorer
- In your Azure Databricks workspace, click Catalog
- Select the catalog you want to publish
- Click the Share button and select Publish to OneLake
- Click Connect to Microsoft Entra ID and authenticate with your account
Step 3: Access in Fabric
- The catalog becomes available in Fabric as a read-only mirrored catalog
- No data movement occurs—credentials are fetched on-demand
Integration Methods Comparison:
| Method | Who Initiates | What Gets Shared | Status |
|–||-|–|
| Publish to Fabric | Azure Databricks catalog admin | Full catalog | Beta |
| Mirror Azure Databricks Catalog | Fabric workspace admin | Selected schemas and tables | Generally Available |
4. Security and Governance Considerations
While the interoperability is powerful, security practitioners must be aware of several critical considerations:
Critical Security Warning: Access to the Azure Databricks Storage item in a Fabric workspace is not restricted to the assigned managed identity. Any principal with a non-viewer role (Member, Contributor, or Administrator) in the Fabric workspace can read, write, and delete data in the Azure Databricks Storage item, regardless of whether they were granted access through Unity Catalog.
Mitigation Strategy:
- Limit the number of principals assigned Member, Contributor, or Administrator roles in that Fabric workspace
- Do not add end users or broad service principals to this workspace unless strictly necessary
- Use a dedicated Fabric workspace solely for the Azure Databricks Storage item—do not mix it with other Fabric assets or user-facing workspaces
Additional Security Considerations:
- When Fabric reads data registered in Unity Catalog, Fabric engines perform authorization using short-lived credentials tied to the identity of the user who configured the connection, not the user querying the data
- Fabric engines do not apply Unity Catalog governance and security policies to Fabric users. When a table is exposed in Fabric, no Unity Catalog-governed access controls apply to Fabric users
- The configuring user requires the `EXTERNAL USE SCHEMA` privilege on the Unity Catalog schema containing the tables accessed by Fabric
- CLI Commands for Managing Unity Catalog and OneLake Integration
For automation and programmatic management, the Databricks CLI provides comprehensive commands:
Install and Configure Databricks CLI:
Install Databricks CLI pip install databricks-cli Configure authentication databricks configure --token
Unity Catalog CLI Commands:
List all catalogs databricks catalogs list List catalogs with limit databricks catalogs list --max-results 10 Create a new catalog databricks catalogs create --1ame myCatalog --comment "My Unity Catalog" List all tables in a catalog databricks tables list --catalog myCatalog --schema mySchema Get table details databricks tables get --full-1ame myCatalog.mySchema.myTable
Manage Storage Credentials:
List storage credentials databricks storage-credentials list Create storage credential databricks storage-credentials create \ --1ame oneLakeCredential \ --type MANAGED_IDENTITY \ --msi-resource-id /subscriptions/.../providers/Microsoft.Databricks/accessConnectors/...
6. Limitations and Known Issues
When implementing this integration, be aware of the following limitations:
- Views Not Supported: Fabric reading data registered in Unity Catalog does not support views, materialized views, or streaming tables
- No Unity Catalog Governance in Fabric: Existing Unity Catalog security policies do not apply to downstream users in Fabric
- Lakehouse Federation Catalogs Not Supported: Lakehouse Federation catalogs cannot be accessed through this integration
- OpenSharing Catalogs Not Supported: Delta Sharing catalogs are not currently supported
- AWS Limitation: Unity Catalog in AWS currently permits storage credentials only for S3-based paths; attempts to use ABFSS for OneLake or ADLS Gen2 are blocked
- Cluster Log Paths: Unity Catalog does not currently support specifying paths using ABFSS for cluster logs
What Undercode Say:
- Key Takeaway 1: The OneLake-Unity Catalog integration represents a paradigm shift from “copy and sync” to “govern once, access everywhere.” Organizations can finally break down data silos without incurring the cost and complexity of data duplication.
- Key Takeaway 2: The security implications cannot be overstated. The current implementation creates a privileged workspace model where Fabric workspace admins effectively bypass Unity Catalog governance. This is a feature, not a bug—but it requires careful planning and strict access control policies.
Analysis: The integration of Unity Catalog with OneLake is a strategic masterstroke that positions Microsoft Fabric and Azure Databricks as complementary rather than competing platforms. For enterprises already invested in Azure, this eliminates the painful “either/or” decision between Databricks’ powerful data engineering capabilities and Fabric’s unified analytics experience. However, the security model—particularly the bypass of Unity Catalog governance when data is accessed through Fabric—creates a new attack surface that security teams must address proactively. The recommendation to use dedicated, restricted workspaces is sound but may limit the agility that makes this integration attractive in the first place. Organizations should treat this as a “privileged access” pattern and implement additional monitoring, logging, and approval workflows around Fabric workspace membership. The Beta and Preview status of key features also means that early adopters should expect changes and potential breaking updates as Microsoft and Databricks refine the integration.
Prediction:
- +1 This integration will accelerate enterprise adoption of both Microsoft Fabric and Azure Databricks, as organizations can now adopt best-of-breed tools without locking themselves into a single vendor’s ecosystem.
- +1 We can expect to see governance enhancements within 12-18 months, including the ability to enforce Unity Catalog policies on Fabric-side access, closing the current security gap.
- -1 The privileged workspace model creates a significant security risk that may deter organizations with strict compliance requirements (finance, healthcare, government) from adopting this integration until the governance model matures.
- +1 The OneLake-Unity Catalog bridge will become the blueprint for future interoperability between cloud data platforms, pressuring competitors like AWS and Google to develop similar cross-platform governance capabilities.
- -1 Organizations that rush to implement this integration without thoroughly understanding the security implications and establishing proper guardrails risk data exposure and compliance violations.
- +1 The elimination of data duplication and complex pipelines will reduce infrastructure costs and operational overhead for data engineering teams by 30-50% in mature implementations.
- +1 This integration positions OneLake as the de facto storage layer for the Microsoft data ecosystem, potentially displacing ADLS Gen2 as the primary storage target for new workloads.
- -1 The complexity of managing identities, permissions, and external locations across two platforms will require new skills and potentially new roles (e.g., “cross-platform data governance specialist”) that are currently in short supply.
- +1 AI practitioners will benefit significantly, as they can now leverage Databricks’ ML capabilities on data stored in OneLake while using Fabric’s Power BI integration for visualization—all without data movement.
- +1 The open API approach to credential vending sets a positive precedent for vendor-1eutral data access, potentially leading to broader industry standards for cross-platform data governance.
▶️ Related Video (76% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Olivierbretondataguru Microsoftfabric – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
