Reduce TypeForm recognition slowdown

[davidfstr]

References #21262

Summary

Enabling TypeForm by default (#21262) regressed
mypy's self-check by ~2% on mypyc-compiled builds.

Details

The regression came entirely
from SemanticAnalyzer.try_parse_as_type_expression, which is now invoked eagerly
on ~2.84M expressions per self-check. Of those, ~3.3% reached the expensive
full-parse block (expr_to_analyzed_type + isolated_error_analysis), and ~91% of
those full parses failed — pure wasted work.

1. This branch adds a series of fast-rejection filters to the offending function (SemanticAnalyzer.try_parse_as_type_expression), eliminating
   ~83% of the full-parse attempts (3,144 → 542) on mypy's self-check, with zero
   correctness regressions. End-to-end this recovers ~43% of the regression by paired
   median (21.6 ms of a 50.2 ms CPU-time regression, n=100, CI ±5 ms; consistent with
   ~34–40% on wall-clock).

2. This branch also adds additional instrumentation to analyze what kinds of
   expressions pass through the fast-rejection filters and make it to a full-parse.
   This instrumentation is enabled via the MYPY_TYPEFORM_PROFILE_FULL_PARSE
   variable and outputs a .tsv file that can be processed using the new
   misc/analyze_typeform_full_parse_profile.py script to classify & aggregate
   expressions.

3. Finally this branch extends misc/perf_compare.py with additional reporting
   options to reduce measured variance.

Each commit on the PR branch has a detailed message giving more information about the specific changes made.

Optimization Results

CPU time — canonical; lowest-variance metric

python misc/perf_compare.py --warmup-runs 3 --num-runs 100 -j 3 \
    --metric cpu --workers1 \
    022d9bc96 16fef2515 HEAD

Commit	Mean	Median	Δ vs baseline (paired median ±95% CI)
022d9bc96 baseline	2.768 s	2.757 s	—
16fef2515 regression	2.817 s	2.809 s	+50.2 ms ±5.2 (+1.82%)
3f39cd753/HEAD branch (all filters)	2.797 s	2.789 s	+28.6 ms ±5.5 (+1.04%)

Details

CPU time (user+sys) on a single worker is the lowest-variance estimator here (CV ≈ 3.3%, vs ≈ 7.5% for wall-clock at the same n) and the truest measure of the per-call work being attacked: with one worker the whole self-check runs serially, so all ~2.84M calls land in one CPU-time figure rather than being spread across workers and hidden behind the slowest one.

The branch recovers 21.6 ms of the 50.2 ms regression — ~43% by paired median
(~40% by trimmed mean), leaving +28.6 ms (~57%, still well outside the ±5 ms CI).
The recovered fraction is the difference of two deltas measured against the same baseline
within one interleaved run, so it inherits the run's low noise (a conservative
independent-error bound is ≈ ±7.6 ms on the 21.6 ms numerator; the true band is tighter
because the two deltas are positively correlated — same baseline, same per-round machine
state).

Wall-clock — default user-facing metric

python misc/perf_compare.py --warmup-runs 3 --num-runs 100 -j 3 \
    022d9bc96 16fef2515 HEAD
python misc/perf_compare.py --warmup-runs 3 --num-runs 400 -j 3 \
    022d9bc96 16fef2515 HEAD

Run	Commit	Median	Δ vs baseline (paired median ±95% CI)
n=100	022d9bc96 baseline	1.342 s	—
	16fef2515 regression	1.369 s	+27.2 ms ±6.7 (+2.03%)
	3f39cd753 branch	1.357 s	+16.3 ms ±6.0 (+1.21%)
n=400	022d9bc96 baseline	1.351 s	—
	16fef2515 regression	1.369 s	+17.6 ms ±3.3 (+1.30%)
	3f39cd753 branch	1.362 s	+11.6 ms ±3.0 (+0.86%)

Details

Wall-clock recovery is **~40% at n=100** (10.9 ms of 27.2 ms) and **~34% at n=400** (6.0 ms of 17.6 ms) by paired median — consistent with the CPU figure once you allow for wall-clock's higher variance and the fact that it is bounded by the slowest worker. The recovered fraction (~34–43% across metrics and run lengths) stays stable even though the absolute regression differs a lot between metrics (≈50 ms CPU vs ≈18–27 ms wall) — the reassuring sign that the recovery is real rather than a noise artifact.

Using the new full-parse instrumentation

Details

# 1. Run instrumented self-check on this branch:
MYPY_TYPEFORM_PROFILE_FULL_PARSE=/tmp/tf.log \
    python -m mypy --config-file mypy_self_check.ini -p mypy --no-incremental

# 2. Aggregate the profile:
python misc/analyze_typeform_full_parse_profile.py --top 20 /tmp/tf.log.*

Open Questions

• Is the MYPY_TYPEFORM_PROFILE_FULL_PARSE env-var name acceptable, or should
  it follow an existing naming convention?

• I did not run performance numbers against non-mypy repositories
  (like PyTorch and Black) as I originally planned. Would you like me to?

• Should the misc/perf_compare.py changes ship in this PR, or land separately?
  They are a general benchmarking-harness improvement (CPU metric, single-worker
  mode, median-based reporting, opt-in binary cache), independent of the TypeForm
  filters — splitting them into their own PR may be cleaner for review.

[ilevkivskyi]

@davidfstr Your original PR was never reverted in master, only in release branch. Hopefully you can still cherry-pick the 4 relevant commits starting from current master.

[davidfstr]

Hopefully you can still cherry-pick the 4 relevant commits starting from current master.

Done. Now there are only 3 commits. The 4th one reapplied enabling TypeForm by default, which is already on master

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

[hauntsaninja]

The right baseline is commits after c0cced3 which optimised this path, not your current baseline

[ilevkivskyi]

Yeah, the best comparison is simply this PR vs master.

In general, I think this (heavy focus on regexps) is a wrong approach. Regexps may give some short term win, but they are themselves slow (when compiled with mypyc), and a pain to maintain. Instead I would propose to start with two things:

• Keep few simple filters that can be expressed without a regexp, like string[0] == "." or string[-1] == "." etc.
• Skip this function completely if the type context is known to be not a TypeForm (e.g. for an inferred variable, or a variable with declared non-TypeForm type etc).