Support None comparisons for null expressions#1489
Open
zeel2104 wants to merge 1 commit intoapache:mainfrom
Open
Support None comparisons for null expressions#1489zeel2104 wants to merge 1 commit intoapache:mainfrom
zeel2104 wants to merge 1 commit intoapache:mainfrom
Conversation
timsaucer
reviewed
Apr 12, 2026
Member
timsaucer
left a comment
timsaucer
left a comment
There was a problem hiding this comment.
Very nice addition. Thank you for the submission! I have one minor recommendation.
Comment on lines
+176
to
+191
| def test_relational_expr_none_uses_null_predicates(): | ||
| ctx = SessionContext() | ||
|
|
||
| batch = pa.RecordBatch.from_arrays( | ||
| [ | ||
| pa.array([1, 2, None]), | ||
| pa.array(["alpha", None, "gamma"], type=pa.string_view()), | ||
| ], | ||
| names=["a", "b"], | ||
| ) | ||
| df = ctx.create_dataframe([[batch]], name="batch_with_nulls") | ||
|
|
||
| assert df.filter(col("a") == None).count() == 1 # noqa: E711 | ||
| assert df.filter(col("a") != None).count() == 2 # noqa: E711 | ||
| assert df.filter(col("b") == None).count() == 1 # noqa: E711 | ||
| assert df.filter(col("b") != None).count() == 2 # noqa: E711 |
Member
There was a problem hiding this comment.
I think you can just update the test_relational_expr to have some null values and incorporate this into the existing test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1483.
Rationale for this change
Comparing expressions to
Nonewith==currently builds a regular equality comparison against a null literal, which follows SQL null semantics and does not match null values in filters. This is surprising for Python users, especially since comparing against other scalar values works as expected and the equivalent.is_null()expression does return the expected rows.What changes are included in this PR?
Expr.__eq__soexpr == Nonemaps toexpr.is_null()Expr.__ne__soexpr != Nonemaps toexpr.is_not_null()== Noneand!= Noneon nullable integer and string columnsAre there any user-facing changes?
Yes. Python users can now write
col("a") == Noneandcol("a") != Noneas shorthand foris_null()andis_not_null().is Noneis not supported because Python identity checks cannot be overloaded.