Azure Databricks Explained

Key takeaways

• ✓Azure Databricks is a managed analytics platform built on Apache Spark, hosted in your Azure environment and billed through Microsoft, making it the default Databricks deployment for Australian enterprises already running Azure infrastructure.

• ✓The platform combines data engineering, machine learning, and SQL analytics on a single lakehouse foundation, removing the need to maintain separate data warehouse and data lake systems.

• ✓Recent features aimed at business users, including Genie in Microsoft Teams and an Excel add-in, let analysts query data in plain English without writing code.

• ✓Pricing is consumption-based (Databricks Units, or DBUs), which keeps entry costs low but can scale quickly without proper governance. Skill gaps are the most common cause of runaway spend.

• ✓Training your team on Azure Databricks is a practical investment: the platform rewards people who understand it and penalises those who don't.

What is Azure Databricks?

Azure Databricks is Databricks running on Microsoft Azure as a fully managed service. Microsoft and Databricks jointly built and support it, which means you get a single product with two vendors standing behind it: Databricks supplies the data and AI platform, and Azure supplies the underlying compute, networking, storage, and identity infrastructure.

In practical terms, your team deploys a Databricks workspace from the Azure portal, pays through your Azure subscription, and the control plane (the orchestration layer that manages clusters, jobs, and users) is run by Databricks. Your data stays in your own Azure storage account, typically Azure Data Lake Storage Gen2. That split matters for governance conversations: Databricks manages the engine, you own the data.

Most Australian enterprises end up on Azure Databricks for straightforward reasons. Many already run Microsoft 365, Azure Active Directory (now Entra ID), and Azure data services such as Synapse or Azure SQL. Databricks slots into that existing footprint without requiring a separate cloud account or a new identity stack. For teams that have already standardised on Azure, adding Databricks is an extension, not a platform shift.

The joint development agreement between Microsoft and Databricks also produces integrations that are tighter than what you typically get between two separately developed products. Genie, Databricks' natural-language data querying feature, has a native integration with Microsoft Teams, meaning a business analyst can ask questions of their data without leaving Teams. There is also a Databricks Excel add-in that lets analysts pull live data from a Databricks SQL warehouse directly into a spreadsheet, which matters in organisations where Excel is still the dominant reporting tool.

If you want the broader picture of how Databricks structures its data and AI platform before getting into Azure-specific detail, the Databricks lakehouse explainer covers the architecture that sits underneath all of this.

How does Azure Databricks fit the lakehouse architecture?

Azure Databricks is built around the lakehouse model, which replaces the older pattern of running a data lake and a data warehouse as two separate systems. If you want more detail on why that split became a problem, the Databricks Lakehouse explained article covers the full story. The short version: teams were maintaining two copies of their data, doing expensive transfers between systems, and still struggling to get a single consistent answer to a business question.

The lakehouse stores everything once, in open file formats on cloud storage (Azure Data Lake Storage Gen2, in this case), and then adds the reliability and query performance that used to require a separate warehouse on top of that same storage.

Delta Lake: the foundation

The layer that makes this work is Delta Lake, an open-source storage format that Databricks created and now governs as part of the Linux Foundation. Delta Lake adds ACID transactions to your cloud storage. ACID (Atomicity, Consistency, Isolation, Durability) is the property that prevents a half-finished write from corrupting a table, or two simultaneous queries from reading inconsistent data. Traditional data lakes on raw object storage had none of this, which is why they needed a warehouse sitting alongside them to handle anything business-critical.

Delta Lake also keeps a transaction log, which gives you a complete history of every change to a table. That history enables time travel: you can query the state of a table as it was yesterday, last week, or three months ago without maintaining separate backups.

Unity Catalog: governance across the whole platform

Data governance sits in Unity Catalog, Databricks' unified metadata and access-control layer. Unity Catalog sits above the compute layer, which means permissions, data lineage, and audit logs apply consistently whether a user is running a Python notebook, a SQL query, or an AI model. In an Azure deployment, Unity Catalog integrates with Azure Active Directory (now Entra ID) for identity, so you are managing access through the same identity provider your organisation already uses.

One catalogue, not three

Before the lakehouse, a typical enterprise might govern access separately in the lake, the warehouse, and a BI tool. Unity Catalog collapses that into one place. A permission change propagates everywhere, which matters when an auditor asks you to demonstrate who had access to a dataset six months ago.

Why this is different from the old architecture

The practical difference is operational load. With a lake-plus-warehouse setup, a data engineering team typically spent a significant portion of its time on pipeline maintenance: extracting data from the lake, transforming it, loading it into the warehouse, and keeping the two in sync. When something broke (and it always broke), tracing the failure meant looking in two places.

On Azure Databricks, the data sits in one place. Transformation, querying, machine learning, and AI workloads all read from and write to the same Delta tables. There is less to maintain, fewer failure points, and a clearer audit trail. The trade-off is that your team needs to understand the platform well enough to design those pipelines correctly from the start, which is where training becomes genuinely important rather than optional.

Which Azure Databricks features matter most for enterprise teams?

Four capability areas account for most of the value enterprises actually extract from Azure Databricks. Each serves a different part of the organisation.

Genie: letting business users query their own data

Genie is Databricks' natural-language interface for data. A business user types a question in plain English, and Genie translates it into SQL, runs it against the lakehouse, and returns a chart or table. No analyst in the loop, no ticket queue.

Two integrations make this practical for Australian enterprise environments. The first is Genie in Microsoft Teams: users stay inside the collaboration tool they already have open, ask a question in a Teams chat, and get a result back without navigating to a separate platform. The second is an Excel add-in that brings the same query capability directly into a spreadsheet, which matters for finance and operations teams who live in Excel and have little interest in learning a new interface.

The quality of Genie's answers depends heavily on how well the underlying data is documented, so the investment in Unity Catalog governance pays off here before users ever type a question.

Genie is only as good as the data behind it

If column names are cryptic, business definitions are missing, or access controls are inconsistent, Genie will produce unreliable answers. Data governance work is a prerequisite, not an afterthought.

Lakeflow: building and managing data pipelines

Lakeflow is the Databricks-native pipeline product. It covers three related problems: ingesting raw data from source systems (Lakeflow Connect), transforming it into clean, analysis-ready tables (Lakeflow Designer and declarative pipelines), and scheduling and monitoring the whole process (Lakeflow Jobs).

For engineering teams, the main advantage is that pipelines run inside the same platform where data is stored and queried. There is no separate orchestration tool to maintain, no data moving between systems for each stage. Errors surface in one place, and lineage (the record of where data came from and how it was transformed) is tracked automatically through Unity Catalog.

Databricks SQL: running analytics at scale

Databricks SQL is a dedicated query engine and BI layer built on top of the lakehouse. Analysts write standard SQL against Delta tables, the Databricks open table format, and get performance comparable to a dedicated data warehouse without the cost of keeping a separate warehouse running alongside the lake.

For teams currently splitting workloads between Azure Data Lake Storage and Azure Synapse Analytics, Databricks SQL is often the consolidation point. The relationship with Power BI is worth understanding here: the two products connect directly, so existing Power BI reports can query Databricks SQL without rebuilding dashboards.

ML and AI workloads

Azure Databricks is where many Australian enterprise teams run machine learning model training, experiment tracking, and, increasingly, generative AI workflows. The MLflow tracking server is built in, which means data scientists can log experiments, compare model versions, and deploy models to production without bolting on additional tooling.

The more recent addition is support for AI agent development through AgentBricks, which allows teams to build, evaluate, and govern AI agents that operate against enterprise data. This is still an emerging pattern, but the infrastructure for it exists inside Azure Databricks today.

How does Azure Databricks pricing work?

Azure Databricks billing comes from two places at once, and understanding both is essential before you approve a workload for production.

Azure infrastructure costs cover the virtual machines, storage, and networking that Microsoft bills through your Azure subscription. These vary by region, VM type, and how long your clusters run.

Databricks platform costs are charged in DBUs (Databricks Units), a measure of processing capacity consumed per hour. DBU rates differ depending on what you are running: interactive clusters used by data engineers cost more per DBU than automated job clusters running overnight pipelines, because interactive compute is priced at a premium.

The tier structure

Databricks on Azure is available in two main tiers:

Most Australian enterprises run Premium. Unity Catalog alone, which handles data governance across workspaces, is a Premium-only feature and one that regulated industries typically cannot do without. You can read more about what Unity Catalog covers in the Unity Catalog explained article.

Where costs tend to surprise teams

The two most common sources of unexpected spend are clusters left running after a job finishes, and interactive clusters sized too large for exploratory work. Databricks provides auto-termination settings, but they only help if someone has configured them correctly.

Skills reduce your bill directly

Teams that understand cluster configuration, autoscaling, and the difference between job clusters and all-purpose clusters consistently run lower DBU spend than teams that use default settings. This is one of the clearest cases where training pays for itself in measurable cost avoidance.

Serverless compute, now available for Databricks SQL and notebooks in supported regions, shifts infrastructure management to Databricks and can reduce idle spend, though the DBU rate is higher. Whether it costs less overall depends on your usage pattern. Bursty, unpredictable workloads often benefit; steady, high-utilisation pipelines often do not.

For a deeper look at how DBUs map to real budget decisions, see the Databricks pricing explained article.

What are the trade-offs compared to other platforms?

Azure Databricks is a strong choice for many enterprise data teams, but it is not the right fit for every situation. Being honest about that matters.

Where Azure Databricks wins

For teams running machine learning pipelines, large-scale data transformation, or building AI applications, Databricks' core Apache Spark engine is hard to beat on throughput and flexibility. The tight integration with Azure Active Directory, Azure DevOps, and Microsoft Fabric means that enterprises already invested in the Microsoft ecosystem can connect Databricks into their existing identity, governance, and BI tooling without a major re-architecture. Unity Catalog adds fine-grained data governance across all workspaces, which matters for regulated industries. And features like Genie in Microsoft Teams and the Excel add-in genuinely lower the barrier for non-technical users to interact with data.

Where it adds complexity

The same depth that makes Databricks powerful also makes it operationally demanding. Cluster configuration, job scheduling, Delta Lake table management, and cost controls all require skilled people to get right. A team without a capable data engineer will spend more time troubleshooting infrastructure than building value. Azure Databricks also adds a billing layer on top of your existing Azure spend, which means pricing can surprise teams that haven't mapped out their DBU consumption in advance.

Compared to Snowflake, Databricks requires more upfront configuration but offers broader support for unstructured data and ML workloads. Snowflake is often easier to stand up quickly and appeals to teams whose primary need is SQL analytics on structured data.

Who it is probably not the right fit for

Small teams running straightforward reporting workflows may find Azure Databricks over-engineered for their needs. If your primary output is dashboards built on clean, structured data, a simpler stack (Azure Synapse Analytics, for instance, or even Power BI Premium with a well-organised data warehouse) will be easier to manage and cheaper to run.

The honest trade-off

Azure Databricks rewards investment. Teams that have the engineering skills to configure it well, and the data volumes to justify the cost, get a platform that scales cleanly. Teams that are still maturing those capabilities may find the complexity outweighs the benefit in the short term.

The skills gap is the most consistent obstacle Australian teams raise. The platform's power is real, but extracting it requires people who understand Spark, Delta Lake, and the Unity Catalog governance model. That is a solvable problem, but it is worth factoring into any adoption plan from the start.

Frequently asked questions

How is Azure Databricks different from Azure Synapse Analytics?

Azure Synapse Analytics is a data warehousing and integration service built primarily around SQL workloads and Azure-native tooling. Azure Databricks is a separate product, built on Apache Spark, that covers a broader range of workloads: data engineering, machine learning, streaming, and AI development. The two platforms overlap in places, and some organisations run both, but Databricks is the stronger choice when teams need serious machine learning capability or are building towards a lakehouse architecture. Synapse tends to suit teams whose work is predominantly SQL-based and who are deeply invested in the Microsoft integration stack.

What is Unity Catalog and do we need it?

Unity Catalog is Databricks' centralised governance layer: it manages access control, data lineage, and auditing across all the data and AI assets in your Databricks environment. For any enterprise operating under regulatory requirements or handling sensitive data, it is effectively essential. It gives you a single place to define who can see what, and it produces the audit trail that compliance teams need. There is a dedicated explanation of how it works in the Unity Catalog guide.

Where does Azure Databricks store data, and can Australian organisations keep it onshore?

Data in Azure Databricks is stored in your own Azure storage account, not in Databricks' infrastructure. That means data residency is determined by the Azure region you configure, not by Databricks. Australian organisations can run their environment in the Australia East or Australia Southeast regions, keeping all data onshore and within the boundaries that Australian Privacy Act obligations and industry-specific requirements may demand. Confirm your specific requirements with your legal and compliance teams, but the architecture does support fully onshore operation.

Can non-technical users actually work with Azure Databricks through tools like Genie?

Yes, and this is one of the more significant recent changes to the platform. Databricks Genie lets business users query data in plain English through a chat interface, without writing SQL. Genie is also available inside Microsoft Teams, so a finance analyst or operations manager can ask questions of their data directly from a tool they already use every day. There is also an Excel add-in that pulls data from the lakehouse into a spreadsheet. These features do not replace the need for data engineers to build and govern the underlying environment, but they meaningfully expand the group of people who can get value from it without specialist training.

How do Australian teams get trained on Azure Databricks?

Most Australian enterprises find that a combination of structured training and hands-on practice works best. Databricks Academy offers official learning paths and certification preparation. For teams who need training built around their actual environment, datasets, and workflows, a custom programme often produces faster, more durable results. Better People delivers Databricks training in Australia as an authorised partner, covering everything from foundational data engineering through to machine learning and AI use cases. The comparison between Databricks Academy and custom training is worth reading if you are deciding which approach suits your team.

How do Australian teams build skills on Azure Databricks?

Most Australian enterprises land in the same place: they have an Azure Databricks environment, a data engineering or analytics team with solid foundations, and a growing gap between what the platform can do and what the team actually uses day to day. Unity Catalog sits largely unconfigured. Genie has not been rolled out to business users. Pipeline monitoring is manual because nobody has had time to learn Lakeflow properly.

That gap is a skills problem, not a technology problem.

The two main paths to closing it are Databricks Academy's self-paced and instructor-led courses, and customised training built around your actual environment and workflows. They serve different needs. Academy content is well-structured and certification-aligned, which makes it a good fit for individuals preparing for the Databricks certifications. Custom training works better when you need a whole team to get productive on your lakehouse, using your Delta tables, Unity Catalog setup, and data sources, without spending weeks on concepts that don't apply to your stack.

The capability gap most teams underestimate

Knowing Databricks in the abstract and knowing how to use it in your specific Azure environment are two different things. Generic training closes the first gap. It rarely closes the second.

There is also the business-user side to consider. If your organisation is rolling out Genie in Teams or the Excel add-in, the people asking questions of your data are not data engineers. They need a different kind of readiness: how to frame a question, how to interpret a result, when to trust the answer and when to ask a data team to verify it. That is not covered in any certification path, and it is often the difference between adoption and abandonment.

For regulated industries, the complexity goes further still. Governance configuration, data residency within the Azure Australia regions, and audit trail requirements all need to be built into how teams are trained, not treated as an afterthought. There is more on that specific challenge in the custom Databricks training for regulated industries content.

If your team is working through any of these gaps, Better People's Databricks training in Australia covers both the technical and the business-user sides, and can be scoped to your Azure environment specifically.