Skip to content

perf(syntax): parse long email input -- 75.2% faster - #199

Open
dexhunter wants to merge 1 commit into
JoshData:mainfrom
dexhunter:perf/email-split-normalization
Open

dexhunter wants to merge 1 commit into
JoshData:mainfrom
dexhunter:perf/email-split-normalization

Conversation

@dexhunter

Copy link
Copy Markdown

TL;DR

  • Optimize split_string_at_unquoted_special when parsing long email input.
  • Skip Unicode normalization when it cannot change parser state, and build the left substring once.
  • Pytest benchmark: the 50,000-character regression takes 61 ms → 15 ms, saving 46 ms per call (75.4% faster).
  • Native validation: pytest tests/test_syntax.py tests/test_main.py tests/test_deliverability.py passes all 344 tests.

What & Why

The splitter normalizes a short window at every character and builds the left substring one character at a time.
Most characters cannot change quote, escape, or delimiter state.

Check normalization only where it can affect those states, then use a slice for ordinary strings.
Keep the existing 32-character window and string-subclass operations so parsing results stay the same.

Validation

Measurements use CPython 3.12 on Linux, with DNS checks disabled for syntax validation.
The regression and ordinary-address rows use pytest call durations, rounded to milliseconds.
Values below 1 ms show no measurable change at that resolution; they do not prove identical latency.

Situation Before After Change
50,000 combining characters followed by @example.com 61 ms 15 ms 46 ms saved; 75.4% faster
Long internationalized-domain regression 1 ms <1 ms Below 1 ms after the change
Ordinary address and dictionary access <1 ms <1 ms No measurable change
58 validation cases, run twice 121.70 ms 30.16 ms 75.2% faster

The repeated workload includes the long input plus short Unicode, quote, escape, display-name, and invalid-address cases.
These statistics summarize three runs of that workload:

Statistic Value
Median new/old time ratio 0.248 (75.2% faster)
95% CI of the median (bootstrap, 10,000 resamples) 0.246–0.249 (75.1–75.4% faster)
Mean ± SD of the ratio 0.248 ± 0.002
P95 ratio (slowest 5% of pairs) 0.250
Number of old/new pairs (n) 27 (3 runs)

To reproduce the repository regression timings:

python -m pytest tests/test_syntax.py -k 'long_input or long_idna' \
  --junitxml=timings.xml -o junit_duration_report=call

The project flake8 and mypy checks also pass.

Why this is safe and easy to maintain

The change keeps one shared state machine and adds three net code lines.
Quote, escape, delimiter, and error behavior stay unchanged.
Existing syntax tests cover Unicode composition, quoted local parts, display names, and long input.

AI assistance

This PR was prepared with AI assistance from Aiden. Candidates were tracked with Weco Observe.

Weco share link: https://dashboard.weco.ai/share/M8ZZgv-1eOo8hzemcc-T9rSBxZiICnvg

Skip normalization when it cannot change parser state and build the left
substring once. Keep the normalization window and shared state machine.

Co-Authored-By: Aiden <aiden@weco.ai>
@JoshData

Copy link
Copy Markdown
Owner

I am not going to have time to review a performance improvement any time soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants