feat: expose array_compact, array_normalize, cosine_distance, inner_product#1567
Open
timsaucer wants to merge 4 commits into
Open
feat: expose array_compact, array_normalize, cosine_distance, inner_product#1567timsaucer wants to merge 4 commits into
timsaucer wants to merge 4 commits into
Conversation
…roduct Adds Python bindings for four scalar functions from datafusion::functions_nested::expr_fn that were not previously surfaced: - array_compact / list_compact: drop NULLs from an array. - array_normalize / list_normalize: L2-normalize a numeric array. - cosine_distance: 1 - cosine_similarity(a, b). - inner_product: dot product of two numeric arrays. Implementation routes each through the existing array_fn! macro in crates/core/src/functions.rs, mirroring the other functions_nested wrappers. Python wrappers in python/datafusion/functions.py follow the established pattern with doctest examples; list_* aliases use the one-line + See Also form per project convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Expand both docstrings with plain-English definitions, worked examples, ranges, use cases, and behavior on edge cases (zero vector → NULL, length-mismatched inputs fail). Adds a zero-vector example to array_normalize and an orthogonal-vector example to cosine_distance. Updates the list_normalize alias summary to match. Co-Authored-By: Claude <noreply@anthropic.com>
Pin the contracts the doctests don't cover: list_compact/list_normalize must produce the same output as their array_* primaries, and cosine_distance/inner_product must reject length-mismatched inputs at execution time.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR exposes four DataFusion 54 nested scalar functions (array/vector helpers) through the Python bindings, rounding out the existing array/vector-distance function surface in datafusion.functions.
Changes:
- Added Rust
_internal(pyo3) bindings forarray_compact,array_normalize,cosine_distance, andinner_product. - Added Python public wrappers +
__all__exports for the above, pluslist_compact/list_normalizealias helpers. - Added unit tests for alias equivalence and length-mismatch error behavior (and doctest coverage via docstrings).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
crates/core/src/functions.rs |
Adds pyo3 wrappers + module registration for the four newly exposed nested functions. |
python/datafusion/functions.py |
Adds public Python wrappers, docstrings/doctests, __all__ exports, and list_* aliases. |
python/tests/test_functions.py |
Adds unit tests validating alias behavior and mismatch-length error cases for vector functions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Match upstream DataFusion SQL alias surface (inner_product UDF registers `dot_product` in its alias list). Also expand `inner_product` docstring with NULL/length-mismatch behavior to match peer distance fns added in this PR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No tracking issue. This was discovered during an audit after upgrading the DataFusion dependency to 54rc1.
Rationale for this change
These four scalar functions exist in
datafusion::functions_nested::expr_fn(DataFusion 54) but were not exposed through the Python bindings. They round out the array/vector-distance surface that callers already see viaarray_distance,array_sort,array_distinct, etc.What changes are included in this PR?
inner_productis the canonical name; the SQL aliasdot_productcontinues to resolve through the upstream UDF's alias list and is mentioned in the docstring.Are there any user-facing changes?
Yes. Six new public functions in
datafusion.functions:array_compact(array)/list_compact(array)array_normalize(array)/list_normalize(array)cosine_distance(array1, array2)inner_product(array1, array2)No breaking changes.