Databricks
Read tables and views from a Databricks SQL warehouse into a Strata flow, and write results to Unity Catalog Delta tables.
Before you start
- A personal access token. Oplema doesn't support OAuth for Databricks.
- The workspace hostname and the SQL warehouse ID or HTTP path. Both are on the warehouse's Connection details tab.
- Access to use the warehouse, and Unity Catalog privileges on the data, as below.
- Connections come from Oplema's servers, not your browser. If the workspace has IP access lists, it must allow them. Reads may also fetch results directly from Databricks-managed storage, so any firewall on that storage matters too. Oplema doesn't promise a fixed IP address; contact us if you need to allowlist.
-- Replace main, sales and the user with your own.
GRANT USE CATALOG ON CATALOG main TO `user@example.com`;
GRANT USE SCHEMA, SELECT ON SCHEMA main.sales TO `user@example.com`;
-- To write:
GRANT MODIFY, CREATE TABLE ON SCHEMA main.sales TO `user@example.com`;Connect
- In Strata, select an Input block to read, or an Output block to write.
- Set Source to Connectors. On an Output block the same control is Destination.
- In the connector dropdown below it, choose Databricks.
- Open the Connection dropdown and choose Create new connection.
- Fill in Workspace hostname (dbc-….cloud.databricks.com), Personal access token and SQL warehouse ID or HTTP path. Azure and Google Cloud hostnames work too. The token is shown as you type.
- Set Name (optional). Without a name, the connection is labeled with the first part of the hostname.
- Select Save connection.
The new connection is selected. Saving only checks the format of what you typed. It doesn't sign in, so wrong details first show up as an error when the pickers below load.
A saved connection can't be edited. To change a password, token or any other field, open the Connection dropdown, choose Delete (the ×) on its row, and create the connection again. Blocks that used the deleted connection must then select the new one.
Read
| Setting | What it does |
|---|---|
| Catalog, Schema, then Table or view | Pick these in order to choose what to read. |
| Columns | All columns or Selected columns. |
| Date column, From and To | Optional. These are inclusive whole days. TIMESTAMP uses UTC. |
Select Refresh data to read again. A few column types need converting in Databricks first; the error names them. Warehouse time is billed by Databricks and may start a stopped warehouse.
Write
Pick Catalog, Schema, then Table. Only ordinary Unity Catalog Delta tables can be written.
- Create new table: enter a Table name, default
output. Names keep their exact spelling. - An existing table: set Write to Replace or Append rows. Replace uses
INSERT OVERWRITEand keeps the table.
Output writes when you run the flow. Choosing tables in the panel never writes. If you pick Create new table and a table with that name already exists, the run fails. Tables with row filters, column masks or inherited policies are refused.
Databricks limits one write to about one million encoded parameter characters, counting names and types. Wide or large outputs fail before anything is written. Reduce rows or columns.
Limits and billing
- Reads: the first 50 MB of each download is free. After that, each download costs $0.01 per GB from your usage. Failed reads aren't charged, and a read stops when your usage runs out.
- Writes: the first 50 MB of each write is free. After that, each write costs $0.01 per GB from your usage, and a write stops when your usage runs out. The parameter limit above caps the size of each write.
- Databricks bills warehouse time separately.
See Billing for who pays and how usage works.
Troubleshooting
| Message | What to do |
|---|---|
Databricks: enter a full workspace hostname or HTTPS origin, not an account-console URL | Use the workspace hostname from the warehouse's connection details. |
Databricks: enter the SQL warehouse ID or /sql/1.0/warehouses/<id> HTTP path | Paste the HTTP path, or just the 16-character ID. |
Databricks: HTTP 401; check token expiry | Create a new token, delete the connection and create it again. |
Databricks: HTTP 403; check warehouse access and data/metadata permissions | Check the warehouse permission and the grants above. If this is a network block, check IP access lists. |
Databricks: external result download failed; check storage access and retry the read | A firewall may be blocking the workspace's storage from Oplema's servers. |
Databricks: external result HTTP <code>; the link may have expired or storage access is blocked | Run the read again. If it repeats, check storage access. |
Databricks: bound parameters exceed Databricks' 1,048,576-character limit; reduce selected rows or columns | Reduce rows or columns before Output. |
Databricks: destination must be an ordinary Unity Catalog Delta table | Choose another table, or create a new one. |
Databricks: loading failed; the new empty table remains. Check it before running again. | Drop the empty table, or select it and run with Append rows. |
Databricks: empty replacement is blocked; existing data was preserved | The incoming table has no rows. Check the flow upstream. |
Databricks: write completion could not be confirmed (statement <id>). Check the destination before running again. | The connection dropped during the write. Look at the table first, because a retry could write the rows twice. |
This transfer exceeds the free 50 MB and the available usage credits | The read or write passed 50 MB and your usage ran out. Add usage, or move less data. |
Prepare your data. Build it in Strata.
Strata opens in your browser, with no account.