Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,5 @@ Dockerfile* text
# crate; regenerated by `npm run build:native`. Tell git/GitHub they're
# machine-generated so they collapse in diffs and are excluded from
# blame and language stats.
native/sea/index.d.ts linguist-generated=true
native/sea/index.js linguist-generated=true
native/kernel/index.d.ts linguist-generated=true
native/kernel/index.js linguist-generated=true
52 changes: 26 additions & 26 deletions .github/workflows/kernel-e2e.yml
Original file line number Diff line number Diff line change
@@ -1,31 +1,31 @@
name: Kernel E2E Tests

# Runs the SEA backend e2e suite (tests/e2e/sea/**) against a real
# Runs the kernel backend e2e suite (tests/e2e/kernel/**) against a real
# Databricks warehouse with a freshly-built napi-rs kernel binding.
#
# The kernel is a private repo with no published binary artifact. We pin
# a kernel SHA in the `KERNEL_REV` file at the repo root, check the kernel
# out via a GitHub App token, and run `npm run build:native` to compile
# the napi binding into native/sea/ in the same checkout the tests run
# the napi binding into native/kernel/ in the same checkout the tests run
# against. Bumping `KERNEL_REV` is the ONLY way to pick up a new kernel
# version — this keeps the driver <-> kernel pair bisectable, so a driver
# change and the kernel revision it depends on always land together.
#
# Why this exists: the committed native/sea/index.d.ts + index.js are the
# Why this exists: the committed native/kernel/index.d.ts + index.js are the
# TypeScript declarations and the napi-rs platform router; the actual
# `.node` binary is gitignored (large, per-platform) and is NOT in the
# repo. The standard `main.yml` e2e job has no binary, so its SEA suite
# repo. The standard `main.yml` e2e job has no binary, so its kernel suite
# skips (it gates on DATABRICKS_PECOTESTING_* secrets it doesn't set).
# This workflow is what actually exercises the SEA path end-to-end against
# This workflow is what actually exercises the kernel path end-to-end against
# a known kernel revision.
#
# Gate semantics:
# - Plain PR events post a synthetic-success check so the required
# "Kernel E2E" check doesn't block PRs that don't touch the SEA path.
# "Kernel E2E" check doesn't block PRs that don't touch the kernel path.
# Real tests run in the merge queue.
# - `kernel-e2e` label triggers a preview run on the PR; the label is
# auto-removed on `synchronize` for the same security reason.
# - merge_group fires the real gate — runs when SEA-relevant files
# - merge_group fires the real gate — runs when kernel-relevant files
# changed, auto-passes otherwise.
#
# Required external setup (one-time, by a repo admin):
Expand Down Expand Up @@ -81,7 +81,7 @@ jobs:

# ───────────────────────────────────────────────────────────────
# Synthetic success on every non-label PR event so the required
# "Kernel E2E" check doesn't permablock PRs that don't touch SEA
# "Kernel E2E" check doesn't permablock PRs that don't touch kernel
# code. Real run happens in the merge queue (or via explicit label).
# ───────────────────────────────────────────────────────────────
skip-kernel-e2e-pr:
Expand Down Expand Up @@ -112,7 +112,7 @@ jobs:
});

# ───────────────────────────────────────────────────────────────
# Detect whether SEA-relevant files changed. Used by both the
# Detect whether kernel-relevant files changed. Used by both the
# labelled-PR path and the merge-queue path to decide between
# "really run the suite" and "auto-pass the check".
# ───────────────────────────────────────────────────────────────
Expand Down Expand Up @@ -148,7 +148,7 @@ jobs:
ref: ${{ steps.refs.outputs.head_sha }}
fetch-depth: 0

- name: Detect SEA-relevant changes
- name: Detect kernel-relevant changes
id: changed
env:
HEAD_SHA: ${{ steps.refs.outputs.head_sha }}
Expand All @@ -157,18 +157,18 @@ jobs:
CHANGED=$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")
echo "Changed files:"
echo "$CHANGED"
# Run when the SEA driver layer, the napi binding contract, SEA
# Run when the kernel driver layer, the napi binding contract, kernel
# e2e tests, this workflow, the kernel revision pin, or core deps
# move.
if echo "$CHANGED" | grep -qE "^(lib/sea/|native/sea/|tests/e2e/sea/|tests/unit/sea/|\.github/workflows/kernel-e2e\.yml|KERNEL_REV|package\.json|package-lock\.json)"; then
if echo "$CHANGED" | grep -qE "^(lib/kernel/|native/kernel/|tests/e2e/kernel/|tests/unit/kernel/|\.github/workflows/kernel-e2e\.yml|KERNEL_REV|package\.json|package-lock\.json)"; then
echo "run_tests=true" >> "$GITHUB_OUTPUT"
else
echo "run_tests=false" >> "$GITHUB_OUTPUT"
fi

# ───────────────────────────────────────────────────────────────
# Real test job. Builds the napi binding from the pinned kernel SHA
# and runs the SEA e2e suite against the dogfood warehouse.
# and runs the kernel e2e suite against the dogfood warehouse.
# ───────────────────────────────────────────────────────────────
run-kernel-e2e:
needs: detect-changes
Expand All @@ -182,7 +182,7 @@ jobs:
checks: write
id-token: write
env:
# SEA e2e tests gate on the DATABRICKS_PECOTESTING_* vars; map the
# kernel e2e tests gate on the DATABRICKS_PECOTESTING_* vars; map the
# warehouse secrets onto them so the suite actually runs (it skips
# when they are absent).
DATABRICKS_PECOTESTING_SERVER_HOSTNAME: ${{ secrets.DATABRICKS_HOST }}
Expand Down Expand Up @@ -273,36 +273,36 @@ jobs:

- name: Build napi binding from pinned kernel
# build:native cd's into ${DATABRICKS_SQL_KERNEL_REPO}/napi, runs the
# napi-rs build, and copies index.* into native/sea/. Pointing it at
# napi-rs build, and copies index.* into native/kernel/. Pointing it at
# the SHA-pinned kernel checkout is what makes the binary match
# KERNEL_REV exactly.
env:
DATABRICKS_SQL_KERNEL_REPO: ${{ github.workspace }}/databricks-sql-kernel
run: npm run build:native

- name: Assert committed binding matches KERNEL_REV
# The committed native/sea/index.d.ts + index.js are the consumer-facing
# The committed native/kernel/index.d.ts + index.js are the consumer-facing
# type contract + platform router; they MUST correspond to the pinned
# kernel. build:native just regenerated them from the KERNEL_REV
# checkout, so any diff means the committed contract drifted from the
# pin — fail loudly and tell the author to commit the regenerated files.
# (The .node binaries are gitignored, so git diff only sees the contract.)
run: |
if ! git diff --exit-code -- native/sea/index.d.ts native/sea/index.js; then
echo "::error::native/sea/index.d.ts / index.js are out of sync with KERNEL_REV ($(tr -d '[:space:]' < KERNEL_REV)). Run 'npm run build:native' against that kernel SHA and commit native/sea/index.*."
if ! git diff --exit-code -- native/kernel/index.d.ts native/kernel/index.js; then
echo "::error::native/kernel/index.d.ts / index.js are out of sync with KERNEL_REV ($(tr -d '[:space:]' < KERNEL_REV)). Run 'npm run build:native' against that kernel SHA and commit native/kernel/index.*."
exit 1
fi
echo "Committed binding matches KERNEL_REV."

- name: Smoke-check binding loads
run: node -e "const b=require('./native/sea'); if(typeof b.version!=='function'){throw new Error('napi binding failed to load')} console.log('kernel binding ok:', b.version())"
run: node -e "const b=require('./native/kernel'); if(typeof b.version!=='function'){throw new Error('napi binding failed to load')} console.log('kernel binding ok:', b.version())"

- name: Run SEA e2e tests
- name: Run kernel e2e tests
# Invoke mocha directly rather than via `npm run e2e -- <glob>`: routing a
# glob through the npm-script's inner shell mangles `**` and silently
# resolves to ZERO files (a false pass). mocha expands the quoted glob
# itself, reliably matching every tests/e2e/sea file.
run: NODE_OPTIONS="--max-old-space-size=4096" npx mocha --config tests/e2e/.mocharc.js "tests/e2e/sea/**/*.test.ts"
# itself, reliably matching every tests/e2e/kernel file.
run: NODE_OPTIONS="--max-old-space-size=4096" npx mocha --config tests/e2e/.mocharc.js "tests/e2e/kernel/**/*.test.ts"

- name: Post Kernel E2E check (success)
if: success()
Expand All @@ -320,7 +320,7 @@ jobs:
completed_at: new Date().toISOString(),
output: {
title: 'Kernel E2E passed',
summary: 'tests/e2e/sea ran green against the pinned kernel SHA.'
summary: 'tests/e2e/kernel ran green against the pinned kernel SHA.'
}
});

Expand All @@ -345,7 +345,7 @@ jobs:
});

# ───────────────────────────────────────────────────────────────
# Auto-pass the Kernel E2E check in the merge queue when no SEA-
# Auto-pass the Kernel E2E check in the merge queue when no kernel-
# relevant files changed.
# ───────────────────────────────────────────────────────────────
auto-pass-merge-queue:
Expand All @@ -371,7 +371,7 @@ jobs:
conclusion: 'success',
completed_at: new Date().toISOString(),
output: {
title: 'Skipped — no SEA-relevant changes',
summary: 'No files under lib/sea/, native/sea/, tests/e2e/sea/, tests/unit/sea/, KERNEL_REV, package.json, or package-lock.json changed.'
title: 'Skipped — no kernel-relevant changes',
summary: 'No files under lib/kernel/, native/kernel/, tests/e2e/kernel/, tests/unit/kernel/, KERNEL_REV, package.json, or package-lock.json changed.'
}
});
10 changes: 5 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ dist
*.DS_Store
lib/version.ts

# SEA native binding — copied/generated from kernel workspace by `npm run build:native`.
# The committed contract is `native/sea/index.d.ts` (TypeScript declarations) and
# `native/sea/index.js` (the napi-rs platform router — small, stable, and required in
# kernel native binding — copied/generated from kernel workspace by `npm run build:native`.
# The committed contract is `native/kernel/index.d.ts` (TypeScript declarations) and
# `native/kernel/index.js` (the napi-rs platform router — small, stable, and required in
# the publish tarball so a missing build step can't ship a tarball that can't load).
# The `.node` binaries are large per-platform artifacts and must NOT be committed;
# in production they arrive via the `@databricks/sql-kernel-<triple>` optional deps.
native/sea/index.node
native/sea/index.*.node
native/kernel/index.node
native/kernel/index.*.node
4 changes: 2 additions & 2 deletions .npmignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
# selects the per-platform `.node` artifact from `@databricks/sql-kernel-*`
# optionalDependencies (populated when the kernel CI publishes them);
# the .d.ts is the consumer-facing type contract.
!native/sea/index.js
!native/sea/index.d.ts
!native/kernel/index.js
!native/kernel/index.d.ts

!LICENSE
!NOTICE
Expand Down
4 changes: 2 additions & 2 deletions .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,5 @@ package-lock.json
# Generated by napi-rs from the kernel's `napi-binding/napi/` crate;
# regenerated by `npm run build:native`. Format follows napi-rs's
# defaults (no semicolons), not this repo's prettier config.
native/sea/index.d.ts
native/sea/index.js
native/kernel/index.d.ts
native/kernel/index.js
8 changes: 4 additions & 4 deletions lib/DBSQLClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ import { buildUserAgentString } from './utils';
import IBackend from './contracts/IBackend';
import { InternalConnectionOptions } from './contracts/InternalConnectionOptions';
import ThriftBackend from './thrift-backend/ThriftBackend';
import SeaBackend from './sea/SeaBackend';
import KernelBackend from './kernel/KernelBackend';
import PlainHttpAuthentication from './connection/auth/PlainHttpAuthentication';
import DatabricksOAuth, { OAuthFlow } from './connection/auth/DatabricksOAuth';
import {
Expand Down Expand Up @@ -627,12 +627,12 @@ export default class DBSQLClient extends EventEmitter implements IDBSQLClient, I

this.connectionProvider = this.createConnectionProvider(options);

// M0: `useSEA` is consumed via a non-exported internal-options cast so it
// M0: `useKernel` is consumed via a non-exported internal-options cast so it
// doesn't ship in the public `.d.ts`. Mirrors Python's `kwargs.get("use_sea")`
// pattern (see databricks-sql-python/src/databricks/sql/session.py).
const internalOptions = options as ConnectionOptions & InternalConnectionOptions;
const backend = internalOptions.useSEA
? new SeaBackend({ context: this })
const backend = internalOptions.useKernel
? new KernelBackend({ context: this })
: new ThriftBackend({
context: this,
onConnectionEvent: (event, payload) => this.forwardConnectionEvent(event, payload),
Expand Down
2 changes: 1 addition & 1 deletion lib/DBSQLOperation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ export default class DBSQLOperation implements IOperation {
/**
* Requests operation status. Returns the Thrift wire response for
* back-compat with existing user code. On the Thrift backend the response
* is returned verbatim; on any other backend (e.g. SEA) the response is
* is returned verbatim; on any other backend (e.g. kernel) the response is
* synthesized from the neutral {@link IOperationBackend.status} result,
* with Thrift-only fields (`taskStatus`, `numModifiedRows`, etc.) left
* undefined.
Expand Down
4 changes: 2 additions & 2 deletions lib/contracts/IBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ import ISessionBackend from './ISessionBackend';

/**
* Top-level backend dispatch handle. One instance per `DBSQLClient`,
* chosen at `connect()` time based on the `useSEA` flag and never
* chosen at `connect()` time based on the `useKernel` flag and never
* re-selected per-call.
*/
export default interface IBackend {
/**
* Establish backend-level state before any session is opened. Implementations
* consume `options` to build backend-specific connection parameters (e.g. the
* SEA backend derives napi-binding `SeaNativeConnectionOptions` from the auth
* kernel backend derives napi-binding `KernelNativeConnectionOptions` from the auth
* + host fields here). Transport-layer connection providers are owned by
* `DBSQLClient` (via `IClientContext`) and exposed to backends through
* constructor injection.
Expand Down
4 changes: 2 additions & 2 deletions lib/contracts/IDBSQLSession.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,13 @@ export type ExecuteStatementOptions = {
*/
queryTags?: Record<string, string | null | undefined>;
/**
* SEA-only: server-side row cap for this statement (kernel `row_limit`). The
* kernel-only: server-side row cap for this statement (kernel `row_limit`). The
* Thrift backend has no execute-time server cap, so this is a no-op there;
* use `maxRows` for the cross-backend client-side fetch limit.
*/
rowLimit?: number;
/**
* SEA-only: per-statement Spark conf overlay (kernel `statement_conf`).
* kernel-only: per-statement Spark conf overlay (kernel `statement_conf`).
* Merged with the serialized `queryTags` (which land under the reserved
* `query_tags` key). Ignored by the Thrift backend.
*/
Expand Down
16 changes: 8 additions & 8 deletions lib/contracts/InternalConnectionOptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
* signature (see `databricks-sql-python/src/databricks/sql/session.py`).
*
* Callers cast `ConnectionOptions` to this type *only* at the read site
* inside the driver; user code that wants to set `useSEA` may still do so
* inside the driver; user code that wants to set `useKernel` may still do so
* via an untyped object literal — the option is not part of the public
* contract and may be removed without notice.
*/
Expand All @@ -17,28 +17,28 @@ export interface InternalConnectionOptions {
* backend instead of the default Thrift backend. Defaults to `false`.
* @internal Not stable; M0 stub only.
*/
useSEA?: boolean;
useKernel?: boolean;

/**
* SEA-only: kernel connection-pool size (`ConnectionOptions.max_connections`).
* kernel-only: kernel connection-pool size (`ConnectionOptions.max_connections`).
* Validated as a positive integer within the napi `u32` range.
* @internal SEA path only.
* @internal kernel path only.
*/
maxConnections?: number;

/**
* SEA-only: verify the server's TLS certificate. Secure-by-default — omit
* kernel-only: verify the server's TLS certificate. Secure-by-default — omit
* to keep full chain + hostname verification; set `false` only to opt into
* the insecure accept-anything mode.
* @internal SEA path only.
* @internal kernel path only.
*/
checkServerCertificate?: boolean;

/**
* SEA-only: PEM-encoded CA certificate (string or `Buffer`) added to the
* kernel-only: PEM-encoded CA certificate (string or `Buffer`) added to the
* trust store on top of the system roots — for TLS-inspecting proxies or
* on-prem internal CAs. Honoured regardless of `checkServerCertificate`.
* @internal SEA path only.
* @internal kernel path only.
*/
customCaCert?: Buffer | string;
}
2 changes: 1 addition & 1 deletion lib/contracts/OperationStatus.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/**
* Backend-neutral operation state. Mirrors the kernel/pyo3 `StatementStatus`
* and the Python connector's `CommandState`, so a SEA `IOperationBackend`
* and the Python connector's `CommandState`, so a kernel `IOperationBackend`
* implementer can return these without depending on the Thrift wire enum.
*
* Thrift mapping (in `ThriftOperationBackend.adaptOperationStatus`):
Expand Down
4 changes: 2 additions & 2 deletions lib/contracts/ResultMetadata.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { TTableSchema } from '../../thrift/TCLIService_types';
/**
* Backend-neutral result-format taxonomy. Mirrors the three on-wire shapes
* `ThriftOperationBackend` actually dispatches on (`COLUMN_BASED_SET`,
* `ARROW_BASED_SET`, `URL_BASED_SET`); a SEA implementer surfaces the same
* `ARROW_BASED_SET`, `URL_BASED_SET`); a kernel implementer surfaces the same
* three so result-handling stays format-agnostic.
*/
export enum ResultFormat {
Expand All @@ -18,7 +18,7 @@ export enum ResultFormat {
* `schema` keeps the Thrift `TTableSchema` shape for now because the public
* `DBSQLOperation.getSchema()` and `getMetadata()` already expose it on
* `IOperation`; carrying it across the boundary preserves back-compat. The
* SEA backend will adapt its column descriptors into the same shape until
* kernel backend will adapt its column descriptors into the same shape until
* the public IOperation surface is migrated in a later PR.
*/
export interface ResultMetadata {
Expand Down
Loading
Loading