Skip to content

Use native AST positions in _parse, drop source-order traversal#535

Merged
Carreau merged 1 commit into
deshaw:masterfrom
Carreau:native-ast-pos
Jun 24, 2026
Merged

Use native AST positions in _parse, drop source-order traversal#535
Carreau merged 1 commit into
deshaw:masterfrom
Carreau:native-ast-pos

Conversation

@Carreau

@Carreau Carreau commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Since Python 3.8 the built-in parser reports correct lineno/col_offset (and end_lineno/end_col_offset) for every node, including multiline string literals, which historically misreported their position as the ending line with col_offset == -1. That made a large chunk of _parse.py obsolete:

  • Replace the recursive _annotate_ast_startpos / _annotate_ast_nodes (which re-derived positions to work around the old quirk) with a flat ast.walk that reads native positions directly, keeping the decorator-start special case.
  • Delete _flatten_ast_nodes, _iter_child_nodes_in_order, _iter_child_nodes_in_order_internal_1 and _walk_ast_nodes_in_order, including the per-Python-version _fields asserts that broke on each new release.
  • Make string_literals / _get_docstring_nodes sort by native startpos instead of relying on the hand-maintained source-order traversal.
  • Remove the dead endpos branch in _split_code_lines (.endpos is never set on AST nodes) and the now-unused imports / AstNodeContext.
  • _autoimp: import MatchAs from ast directly rather than re-exporting it through _parse.

Net -335 lines in _parse.py with no behavior change on Python >=3.12 (the only version-specific path, the <3.12 f-string clamp, never ran there). All parse/import/autoimp suites pass.

Since Python 3.8 the built-in parser reports correct lineno/col_offset
(and end_lineno/end_col_offset) for every node, including multiline
string literals, which historically misreported their position as the
ending line with col_offset == -1. That made a large chunk of _parse.py
obsolete:

- Replace the recursive _annotate_ast_startpos / _annotate_ast_nodes
  (which re-derived positions to work around the old quirk) with a flat
  ast.walk that reads native positions directly, keeping the
  decorator-start special case.
- Delete _flatten_ast_nodes, _iter_child_nodes_in_order,
  _iter_child_nodes_in_order_internal_1 and _walk_ast_nodes_in_order,
  including the per-Python-version _fields asserts that broke on each
  new release.
- Make string_literals / _get_docstring_nodes sort by native startpos
  instead of relying on the hand-maintained source-order traversal.
- Remove the dead endpos branch in _split_code_lines (.endpos is never
  set on AST nodes) and the now-unused imports / AstNodeContext.
- _autoimp: import MatchAs from ast directly rather than re-exporting it
  through _parse.

Net -335 lines in _parse.py with no behavior change on Python >=3.12
(the only version-specific path, the <3.12 f-string clamp, never ran
there). All parse/import/autoimp suites pass.
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.97%. Comparing base (ebbe1b4) to head (3828e29).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #535      +/-   ##
==========================================
+ Coverage   89.92%   89.97%   +0.04%     
==========================================
  Files          57       57              
  Lines       17694    17580     -114     
==========================================
- Hits        15912    15818      -94     
+ Misses       1782     1762      -20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Carreau Carreau merged commit 1a2a68a into deshaw:master Jun 24, 2026
26 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant