---
title: "Querying Apache Iceberg tables on Databricks with Unity Catalog — Datagist"
description: "An architect's guide to Iceberg on Databricks: managed, foreign and Delta tables, the Iceberg REST catalog, cross-engine access control and limits."
url: https://datagist.ai/insights/query-iceberg-tables-databricks-borderless-lakehouse
---

[Insights](https://datagist.ai/insights) Article

# Querying Apache Iceberg tables on Databricks for a borderless lakehouse

How Unity Catalog serves Apache Iceberg tables to Databricks, Snowflake, Spark, Trino and other engines from one copy of the data, and the table types, access rules and limits to plan around.

Datagist November 3, 2026 7 min read

[Databricks](https://datagist.ai/practices/databricks) [Modern Data](https://datagist.ai/services/modern-data) [Governance & Operations](https://datagist.ai/services/governance-operations)

Few companies run their data on one engine. Data engineers build pipelines in Databricks. Finance reports from Snowflake. A data science team reaches for Trino, DuckDB or plain Python. The usual fix is to copy tables from one platform to the next every night. Each copy needs its own pipeline, its own permissions and its own storage bill, and before long the copies disagree.

Apache Iceberg is an open table format that many engines can read and write. On its own, though, a format only describes files. Engines also need a catalog that says where each table lives, which snapshot is current and who may read it. Unity Catalog can now play that part for Iceberg. Databricks reads and writes Iceberg tables natively, and other engines reach the same tables through Unity Catalog’s Iceberg REST catalog. One table in storage, one set of grants and many engines on top is what we mean by a borderless lakehouse.

This article covers the three kinds of Iceberg table in Unity Catalog, how outside engines connect, how access control carries across, and the limits worth knowing before you design around it.

## Three kinds of Iceberg table in Unity Catalog

Managed Iceberg tables are owned by Unity Catalog. It controls where the files live, commits every change and runs the upkeep: compaction, snapshot expiry and, if you turn it on, liquid clustering. You create one in Databricks SQL by adding `USING ICEBERG` to `CREATE TABLE`. Leave that out and Databricks creates a Delta table, so put it in your table templates. Managed Iceberg needs Databricks Runtime 16.4 LTS or above, serverless compute and predictive optimization turned on in the workspace.

Delta tables with Iceberg reads, also known as UniForm, stay Delta tables. After each commit, Databricks writes Iceberg metadata next to the Delta log, so Iceberg clients can read the table without a second copy of the data. Those clients can only read. This is the low-risk choice when your pipelines already write Delta and you only need another engine to query the results.

Foreign Iceberg tables stay registered in another catalog, such as AWS Glue, a Hive metastore or Snowflake Horizon Catalog. Databricks connects to that catalog through Lakehouse Federation, reads the metadata and then reads the files straight from storage. Foreign tables are read-only in Databricks and get fewer platform features than managed ones. They are a good way to query data in place while you decide what to move.

| Table type               | Who owns the metadata                                                            | Databricks     | Other Iceberg engines                   | Use it when                                                          |
| ------------------------ | -------------------------------------------------------------------------------- | -------------- | --------------------------------------- | -------------------------------------------------------------------- |
| Managed Iceberg          | Unity Catalog                                                                    | Read and write | Read and write through the REST catalog | Several engines need to work on the same table                       |
| Delta with Iceberg reads | Unity Catalog (Delta, with Iceberg metadata added)                               | Read and write | Read only                               | Delta pipelines stay as they are and other engines only read         |
| Foreign Iceberg          | Another catalog, such as AWS Glue, a Hive metastore or Snowflake Horizon Catalog | Read only      | Read only                               | You want to query tables where they are before deciding to move them |

## How other engines connect

Unity Catalog serves the Iceberg REST catalog API at `/api/2.1/unity-catalog/iceberg-rest` on your workspace URL. An engine points its Iceberg catalog at that address and names a Unity Catalog catalog as its warehouse. Schemas then show up as namespaces and tables as Iceberg tables.

Three things have to be in place first. A metastore admin turns on external data access for the metastore. The principal the engine uses, ideally a service principal, gets `EXTERNAL USE SCHEMA` on each schema it should reach, along with the usual `SELECT` or `MODIFY`. And the engine signs in with OAuth or a personal access token. Use OAuth with a service principal for anything that runs on a schedule.

With credential vending, Unity Catalog hands the engine short-lived storage credentials scoped to what that principal may do. They expire after an hour by default. The engine never holds long-lived keys to your bucket, and once you remove a grant, Unity Catalog stops issuing new credentials for that table. Credential vending doesn’t cover foreign tables, so engines reading those still need their own storage access.

For Apache Spark outside Databricks, add the Iceberg Spark runtime and your cloud’s bundle, then set a catalog with type `rest`, the endpoint as its URI and the Unity Catalog catalog name as the warehouse. On AWS, set the region as well, or table scans fail. For Snowflake, create a catalog integration with `CATALOG_SOURCE = ICEBERG_REST` and vended credentials. From there you can link a whole Snowflake database to the Unity Catalog catalog, which keeps the table list in sync, or register single tables with `CREATE ICEBERG TABLE` and refresh them yourself. PyIceberg, Trino, Flink, DuckDB and other Iceberg clients use the same endpoint. Databricks recommends Iceberg client version 1.9.2 or above.

[One copy of the data, many engines. Iceberg tables sit in cloud storage and are registered in Unity Catalog. Databricks reads and writes them directly. Apache Spark, Snowflake, Trino, PyIceberg and other Iceberg clients connect to the Unity Catalog Iceberg REST catalog, which checks each principal's grants and hands out short-lived storage credentials. Managed Iceberg tables accept reads and writes from these engines. Delta tables with Iceberg reads and foreign Iceberg tables from AWS Glue, a Hive metastore or Snowflake Horizon Catalog are read-only for them.](https://datagist.ai/media/img/5948fa9f84f91b86bf2ae2c71d2a04761c817883-1520x1000-96b27e96f7.webp)

## Access control across engines

Because every engine goes through Unity Catalog, table grants apply the same way in Snowflake or Spark as in Databricks. That is the main reason to use the REST catalog instead of pointing engines at storage paths. Path-based access to managed tables isn’t supported anyway.

Row filters and column masks need more care. Databricks documents a way for external engines to read tables that have row filters, column masks or attribute-based policies. A serverless layer in Databricks applies the policy and sends back only the rows and values the reader may see. This works for reads only, so any principal that writes to the table has to be exempt from the policy. It needs recent clients, with the documented Spark setup on Iceberg 1.11 and Spark 4.0 or above, and the serverless work is billed. Test it with the exact engines and versions you run before you count on it for regulated data.

External engines can’t read Unity Catalog views, and dynamic views aren’t supported for them either. If your security model today relies on views to hide rows or columns, move those rules into row filters and column masks on the tables themselves before you open the tables to other engines.

## Writing from more than one engine

Managed Iceberg tables accept writes from outside engines through the REST catalog. Unity Catalog commits each change and rejects commits that would add the same data file twice. A few rules still apply to those writers. They can’t run upkeep such as expiring snapshots or removing orphan files, because Unity Catalog does that. They can’t commit equality deletes, so streaming upsert jobs that use them, which is common with Flink, need to switch to copy-on-write, or to deletion vectors on Iceberg v3\. And they can’t create tables with UUID, fixed, TIME or some nested struct columns.

Even with commits handled safely, give each table one writing engine. Two engines writing to the same table on separate schedules can conflict and retry, and when a number looks wrong you want one pipeline to check. Let other engines read and do their own work in their own tables.

## Iceberg v3

Databricks supports Iceberg v3 on Databricks Runtime 18 LTS and above. v3 brings deletion vectors, which are on by default for new v3 tables and make updates and deletes much cheaper. It also adds the VARIANT type for semi-structured data, row lineage, geometry and geography types, and column write defaults. You can create a v3 table directly or upgrade one by setting its format version.

Check every engine that reads the table before you upgrade. An engine that only understands v2 can’t read a v3 table, and going back means restoring an earlier version. Upgrade a table only when its readers are ready for v3.

## Limits to plan around

Managed Iceberg tables don’t support generated columns, constraints or collations. Branches and tags aren’t supported. Partition evolution works only from external engines, not Databricks SQL, so for new tables lean on liquid clustering instead of a partition scheme you expect to change. The compression codec is fixed at Zstandard.

Foreign tables aren’t refreshed automatically for engines reading them through the REST catalog, so run `REFRESH FOREIGN TABLE` after the source changes. Time travel on a foreign table only reaches snapshots Databricks has already read.

On Delta tables with Iceberg reads, the Iceberg metadata is written after the Delta commit. With very frequent commits, several Delta versions can fold into one Iceberg snapshot, so Iceberg readers may trail Delta readers by a little. Delta deletion vectors can’t be used while Iceberg v2 reads are on.

## Where to start

Pick one table that you already copy between two platforms. A good candidate is an orders or customer table that Databricks builds and Snowflake loads every night. If Databricks stays the only writer and the table is Delta today, turn on Iceberg reads. If more than one engine needs to write, or you are building the table new, make it managed Iceberg.

Connect the second engine through the REST catalog with a service principal and vended credentials. Run both paths side by side for a couple of weeks and compare row counts and totals. Once they match, switch the readers over and retire the copy job. Then move any view-based security into row filters and column masks, and repeat for the next table that two teams copy today.

Tables registered in Glue or a Hive metastore can follow the same path. Federate them as foreign tables first, query them in place, and move the busiest ones to managed Iceberg when you want Unity Catalog to handle upkeep and writes.

---

We design and build lakehouse platforms on [Databricks](https://datagist.ai/practices/databricks), including setups where Snowflake and other engines share the same tables. If you are planning to open your Unity Catalog tables to other engines, [talk to us](https://datagist.ai/contact) about table types, access rules and the order to move things in.
