feat(backend/kernel): match Thrift backend's user-visible surface#793
Merged
Conversation
Three small parity fixes that close 34 of the 66 diffs the Thrift vs Kernel comparator surfaced against Dogfood today (see databricks-driver-test docs/comparator-thrift-vs-kernel-diffs.md). None of these are user-data correctness — they're contract-level differences that affect code reading the cursor's description, catching server errors by class, or calling get_columns with no catalog. 1. PEP-249 description.null_ok always None description_from_arrow_schema previously took null_ok from the Arrow field.nullable bit. The Thrift backend has always reported None for this slot (PEP 249 permits either). Hardcode None so the kernel backend is a drop-in for code that reads cursor.description[i][6]. 2. SqlError maps to ServerOperationError Server-side SQL failures (syntax error, missing object, etc.) used to wrap into the generic DatabaseError on the kernel backend. The Thrift backend raises ServerOperationError for the same shape. ServerOperationError is a DatabaseError subclass, so existing catches of the base class are unaffected — but code that catches the specific subclass now works equivalently. 3. get_columns accepts catalog_name=None Previously rejected at the connector layer with a ProgrammingError. The kernel's list_columns now issues SHOW COLUMNS IN ALL CATALOGS server-side when catalog is None; the response carries catalogName per row so TABLE_CAT is correctly attributed without client-side enumeration. Matches Thrift's getColumns(null, ...) behaviour. Two other comparator-surfaced gaps are out of scope: - INTERVAL columns crashing on fetch (kernel-side fix; intervals need to be stringified post-decode). - _use_arrow_native_complex_types=False not honoured (pre-existing gap shared with the native SEA backend; needs separate work). Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
The kernel writes the server-reported type name into Arrow field metadata under `databricks.type_name` (see `databricks_sql_kernel::reader::metadata_keys::TYPE_NAME`). `description_from_arrow_schema` now consults it and emits 'variant' when the metadata says so, matching the Thrift backend. Today only VARIANT needs this remap — every other precise type either lands on a dedicated Arrow shape (INT, DECIMAL, …) or collapses on both backends (INTERVAL_*, GEOMETRY, GEOGRAPHY). Closes 3 PREPARED_STATEMENT_TYPES / COMPLEX_TYPES diffs in the comparator. Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
Contributor
msrathore-db
left a comment
msrathore-db
left a comment
There was a problem hiding this comment.
LGTM other than the 2 comments
msrathore-db
approved these changes
May 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three small parity fixes that close 34 of the 66 diffs the Thrift vs Kernel comparator surfaced against Dogfood today (see
databricks-driver-test#303for the full diff catalog). None of these are data-correctness changes — they're contract-level differences that affect code reading the cursor's description, catching server errors by class, or callingget_columnswith no catalog.1. PEP-249
description.null_okis alwaysNone(closes 29 diffs)description_from_arrow_schemapreviously tooknull_okfrom the Arrowfield.nullablebit. The Thrift backend has always reportedNonefor this slot (PEP 249 permits either). HardcodeNoneso the kernel backend is a drop-in for code that readscursor.description[i][6].2.
SqlErrormaps toServerOperationError(closes 3 diffs)Server-side SQL failures (syntax error, missing object, etc.) used to wrap into the generic
DatabaseErroron the kernel backend. The Thrift backend raisesServerOperationErrorfor the same shape.ServerOperationErroris aDatabaseErrorsubclass, so existing catches of the base class are unaffected — but code that catches the specific subclass now works equivalently.3.
get_columns(catalog_name=None)accepted (closes 2 diffs)Previously rejected at the connector layer with a
ProgrammingError. The kernel'slist_columnsnow issuesSHOW COLUMNS IN ALL CATALOGSserver-side when catalog isNone(perdatabricks-sql-kernel#33); the response carriescatalogNameper row soTABLE_CATis correctly attributed without client-side enumeration. Matches Thrift'sgetColumns(null, …)behaviour.Out of scope (separate PRs)
pyarrow.compute.cast(month_interval → utf8)is not implemented. Separate kernel PR._use_arrow_native_complex_types=Falsenot honoured (2 diffs) — pre-existing gap shared with the native SEA backend; needs separate work that fixes both.Test plan
python -m pytest tests/unit/— 715 passed, 4 skipped (matches prior baseline; updated kernel-backend tests to reflect the new contract).black src/databricks/sql/backend/kernel/ tests/unit/test_kernel_*.pyclean.This pull request and its description were written by Claude Code.