feat(kernel): wire retry/backoff params + verify metric-view on use_kernel#820
Open
vikrantpuppala wants to merge 1 commit into
Open
feat(kernel): wire retry/backoff params + verify metric-view on use_kernel#820vikrantpuppala wants to merge 1 commit into
vikrantpuppala wants to merge 1 commit into
Conversation
…-view passthrough Consume the kernel's newly-configurable retry policy (kernel PR #117) from the use_kernel path, and confirm UC Metric View metadata works end-to-end now that the kernel allowlists its session conf. Bumps KERNEL_REV to the merged kernel main. Retry: - session.py forwards the connector's _retry_* kwargs into the kernel client (new retry_options, kernel-only; Thrift/SEA unaffected). - client._kernel_retry_kwargs maps them to the kernel Session retry kwargs: _retry_delay_min -> retry_min_wait_secs, _retry_delay_max -> retry_max_wait_secs, _retry_stop_after_attempts_count -> retry_max_attempts (1:1 total-attempts; the kernel does the retries-after-first conversion), _retry_stop_after_attempts_duration -> retry_overall_timeout_secs. Connector delays are float seconds; the kernel takes whole seconds, so we round, flooring any positive sub-second value to 1s (never collapse a configured backoff to 0). _retry_delay_default has no kernel equivalent (the kernel's no-Retry-After backoff is exponential from retry_min_wait). Metric View: - No connector wiring needed: enable_metric_view_metadata already injects spark.sql.thriftserver.metadata.metricview.enabled into session_configuration (backend-agnostic), which the kernel client passes through to session_conf. Kernel #117 now allowlists that conf verbatim, so it reaches the server on the use_kernel path. Added an e2e test confirming the round-trip. KERNEL_REV -> b4d88220cdfad8dba1cfa89892269342ae26feeb (kernel main with retry config + metric-view allowlist). Tests: unit tests for _kernel_retry_kwargs (rounding, sub-second floor, count 1:1, only-set-keys) and retry-options threading through session.py (wheel-independent, verified with the kernel import blocked); live e2e (retry params accepted + metric-view passthrough) green against dogfood. black + mypy clean. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
msrathore-db
approved these changes
Jun 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Connector follow-up to kernel PR #117 (configurable retry/backoff + UC Metric View session-conf allowlist). Wires the connector's retry tuning to the kernel path and confirms metric-view metadata works end-to-end. Bumps
KERNEL_REVto the merged kernel main.Retry/backoff
The kernel owns the retry loop on the
use_kernelpath. This forwards the connector's existing_retry_*kwargs to it:session.pyadds aretry_optionsdict to the kernel client (kernel-only; Thrift/SEA unaffected).client._kernel_retry_kwargsmaps them to the kernelSessionretry kwargs:_retry_delay_minretry_min_wait_secs_retry_delay_maxretry_max_wait_secs_retry_stop_after_attempts_countretry_max_attempts_retry_stop_after_attempts_durationretry_overall_timeout_secs_retry_delay_defaultretry_min_wait)Connector delays are float seconds; the kernel takes whole seconds. We round, flooring any positive sub-second value to 1s so a configured backoff never collapses to "no wait".
Metric View — no wiring needed
enable_metric_view_metadata=Truealready injectsspark.sql.thriftserver.metadata.metricview.enabledintosession_configuration(backend-agnostic, before backend selection), and the kernel client passessession_configuration→session_confverbatim. Kernel #117 now allowlists that conf (sent verbatim, not uppercased —spark.*confs are case-sensitive), so it reaches the server onuse_kernel. Added an e2e test for the round-trip.Verification
_kernel_retry_kwargsmapping (rounding, sub-second floor, count 1:1, only-set-keys) + retry-options threading throughsession.py(wheel-independent; verified withimport databricks_sql_kernelblocked, reproducing the no-wheel CI).black+mypyclean.KERNEL_REV→b4d88220cdfad8dba1cfa89892269342ae26feeb.This pull request and its description were written by Isaac.