Databricks File Federation stopped working — generic permission error despite successful authentication and catalog access

We are setting up a Databricks File Federation connection from Salesforce Data 360 to Azure Databricks / Unity Catalog.

The connection uses:

• Authentication: Identity Provider Based
• Service Principal ID
• Storage Type: CATALOG_PROVIDED

The Salesforce - Databricks connection itself appears to be working correctly:

• Salesforce can authenticate to Databricks.
• Salesforce can discover the Unity Catalog schemas and tables.
• The service principal has the required Unity Catalog permissions.
• The same service principal can successfully access the Databricks Iceberg REST endpoint directly via Databricks CLI.

However, when creating/validating a File Federation data stream, Salesforce reports: 

The user does not have necessary permission to access the object: demo_customer in schema: silver in database: demo_catalog

We have verified the relevant Unity Catalog permissions and reproduced the issue with a newly recreated table. 

For reference, we're using a Databricks Serverless compute. 

The particularly unusual part is that the same notebook/code previously produced a table that worked with File Federation a few days earlier. The same setup worked on the 31st of August, but didn't work on the 4th of September. We've tried with different tables to no avail.

The table is a managed Delta table with Databricks Iceberg compatibility enabled:

• delta.enableIcebergCompatV2 = true
• delta.universalFormat.enabledFormats = iceberg
• delta.enableDeletionVectors = false
• delta.enableRowTracking = true
• delta.columnMapping.mode = name

We have also tested other table configurations and verified that the issue is not resolved by changing the Unity Catalog grants. All necessary permissions are in place.

I also checked Databricks audit logs related to interaction with Salesforce and I could verify: 

- Auth succeeds (oidcTokenAuthorization, mintOAuthToken) — 200

- listNamespaces, listSchemas, listTables on vwfs_dbx_salesforce.silver — 200, all resolved

- getTable for vwfs_dbx_salesforce.silver.demo_customer — 200, resolves to table_id

- authzEval explicitly checking SelectTable on that exact table ID — 200, access granted

- generateTemporaryTableCredential — 200, and it actually issued a real, usable SAS-backed credential and abfss:// URL

- uniformIcebergRestCatalog loadTable — 200

, the Iceberg REST catalog successfully served the table 

So nothing failed from Databricks side, it provided what it needed to. I guess it fails when Salesforce tries to read the files from storage after receiving the temporary credential. But nothing changed security or network wise in the storage account. We are using a managed databricks storage account and vnet and have been since the start when it worked.  

I then tried isolating the issue by creating a new Azure Storage Account that I control, with public network access and no firewall restrictions, and set up Azure Storage diagnostic logging.

I created a simple Iceberg V2-compatible table there and tested it through Salesforce File Federation.

The logs show that Salesforce does reach the storage account and successfully reads the Iceberg metadata JSON (GetBlob→ 206 Success, authenticated via OAuth).

However, that's the only activity I see. Salesforce does not subsequently request the manifest .avro file or any Parquet data files before returning the same "user does not have necessary permission" error.