Conversation
Expose upstream DataFusion DataFrame methods that were not yet available in the Python API. Closes apache#1455. Set operations: - except_distinct: set difference with deduplication - intersect_distinct: set intersection with deduplication - union_by_name: union matching columns by name instead of position - union_by_name_distinct: union by name with deduplication Query: - distinct_on: deduplicate rows based on specific columns - sort_by: sort by expressions with ascending order and nulls last Note: show_limit is already covered by the existing show(num) method. explain_with_options and with_param_values are deferred as they require exposing additional types (ExplainOption, ParamValues). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extend the existing explain() method with an optional format parameter instead of adding a separate explain_with_options() method. This keeps the API simple while exposing all upstream ExplainOption functionality. Available formats: indent (default), tree, pgjson, graphviz. The ExplainFormat enum is exported from the top-level datafusion module. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expose remaining DataFrame methods from upstream DataFusion. Closes apache#1456. - window(*exprs): apply window function expressions and append results as new columns - unnest_column/unnest_columns: add optional recursions parameter for controlling unnest depth via (input_column, output_column, depth) tuples Note: drop_columns is already exposed as the existing drop() method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR extends the Python DataFrame API to expose additional upstream DataFusion DataFrame functionality (set operations, explain formatting, window expressions, sort convenience, and configurable unnesting) and adds corresponding unit tests.
Changes:
- Added
ExplainFormatand extendedDataFrame.explain()to support configurable output formats. - Exposed additional DataFrame operations to Python:
window,sort_by,distinct_on,except_distinct,intersect_distinct,union_by_name, andunion_by_name_distinct. - Extended
unnest_column(s)to accept optionalrecursionsand added tests for the new APIs.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
python/tests/test_dataframe.py |
Adds unit tests covering newly exposed DataFrame methods and explain formatting. |
python/datafusion/dataframe.py |
Adds ExplainFormat and new/extended DataFrame methods (window, set ops, distinct_on, sort_by, unnest recursion options). |
python/datafusion/__init__.py |
Re-exports ExplainFormat from the top-level package API. |
crates/core/src/dataframe.rs |
Adds/extends PyO3 bindings for the new DataFrame methods and explain formatting + unnest recursion options. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| This is a convenience method that sorts all columns in ascending order | ||
| with nulls last. For more control over sort direction and null ordering, | ||
| use :py:meth:`sort` instead. |
There was a problem hiding this comment.
The sort_by docstring says this method "sorts all columns". The implementation actually sorts by the provided exprs only; it does not default to sorting by every column in the DataFrame. Please adjust the wording to avoid implying it sorts the entire schema by default (e.g., "sorts the DataFrame by the given expressions in ascending order with nulls last").
| This is a convenience method that sorts all columns in ascending order | |
| with nulls last. For more control over sort direction and null ordering, | |
| use :py:meth:`sort` instead. | |
| This is a convenience method that sorts the DataFrame by the given | |
| expressions in ascending order with nulls last. For more control over | |
| sort direction and null ordering, use :py:meth:`sort` instead. |
| def except_distinct(self, other: DataFrame) -> DataFrame: | ||
| """Calculate the set difference with deduplication. | ||
|
|
||
| Returns rows that are in this DataFrame but not in ``other``, | ||
| removing any duplicates. This is the complement of :py:meth:`except_all` | ||
| which preserves duplicates. | ||
|
|
There was a problem hiding this comment.
This PR description indicates it closes #1455/#1456 (which include exposing DataFrame.with_param_values), but with_param_values still isn't present on the Python DataFrame API (and isn't bound in crates/core/src/dataframe.rs). Either expose with_param_values on DataFrame as well, or update the PR metadata/issues being closed so they accurately reflect what's implemented here.
Which issue does this PR close?
Closes #1455
Closes #1456
Rationale for this change
These features exist in the upstream repository but have not been exposed in Python.
What changes are included in this PR?
Exposes functions to python
Adds unit tests
Are there any user-facing changes?
Addition only