Skip to content

Commit 8023d08

Browse files
jasmith-hsclaude
jasmith-hs
and
claude
committed
Add autoresearch prompt and benchmark runner for jinjava-3.0 perf work
jinjava-3.0 regressed ~30% on complex template rendering vs 2.8.x due to security hardening. These files set up the framework for systematic optimization: a benchmark runner script and an autoresearch prompt with baseline numbers, hotpath analysis, and optimization ideas. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent ae26ce2 commit 8023d08

2 files changed

Lines changed: 181 additions & 0 deletions

File tree

‎auto/autoresearch.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Jinjava 3.0 Performance Optimization — Autoresearch Prompt
2+
3+
## Objective
4+
5+
Optimize Jinjava 3.0 rendering throughput (ops/s) as measured by JMH benchmarks. The jinjava-3.0 branch introduced security hardening (sandboxed class resolution, method validation) that regressed template rendering performance by ~30% on complex templates and ~8% on simple templates compared to 2.8.x. The goal is to recover as much of that throughput as possible without weakening the security model.
6+
7+
## How to Run
8+
9+
```bash
10+
bash auto/autoresearch.sh
11+
```
12+
13+
Or manually:
14+
```bash
15+
mvn test -q # correctness gate
16+
mvn test-compile -q # compile benchmarks
17+
CP_FILE=$(mktemp)
18+
mvn dependency:build-classpath -Dmdep.includeScope=test -Dmdep.outputFile="$CP_FILE" -q
19+
java -cp "$(cat "$CP_FILE"):target/classes:target/test-classes" \
20+
com.hubspot.jinjava.benchmarks.JinjavaBenchmark
21+
rm -f "$CP_FILE"
22+
```
23+
24+
## Metrics
25+
26+
All benchmarks: `src/test/java/com/hubspot/jinjava/benchmarks/JinjavaBenchmark.java`
27+
28+
**Primary** (these dominate real-world usage):
29+
- `complexTemplateBenchmark` — 20 articles with nested loops, conditionals, filters
30+
- `simpleTemplateBenchmark` — navigation + table with for-loops
31+
32+
**Secondary:**
33+
- `simpleExpressionBenchmark` — single `{{ name }}` substitution
34+
- `filterChainBenchmark` — `capitalize|replace|trim` chain
35+
- `jinjavaCreationBenchmark` — `new Jinjava()` cost
36+
- `jinjavaWithConfigCreationBenchmark` — `new Jinjava(config)` cost
37+
38+
## Render Pipeline
39+
40+
```
41+
Jinjava.render(template, bindings)
42+
└─ Jinjava.renderForResult()
43+
├─ copyGlobalContext() ← copies tags/filters/functions/expTests
44+
├─ new Context(global, bindings)
45+
└─ JinjavaInterpreter.render()
46+
├─ TreeParser.parse() ← tokenize → AST (Node tree)
47+
│ └─ tokenize() ← scan for {{ }}, {% %}, {# #}
48+
├─ Walk Node tree:
49+
│ ├─ TextNode → append literal text
50+
│ ├─ ExpressionNode → ExpressionResolver.resolve()
51+
│ │ └─ JUEL EL evaluation
52+
│ └─ TagNode → Tag.interpret()
53+
│ ├─ ForTag ← dominates template benchmarks
54+
│ ├─ IfTag
55+
│ └─ SetTag, etc.
56+
└─ OutputList.getValue() ← assemble final string
57+
```
58+
59+
## Key Hotpath Files
60+
61+
| File | Role |
62+
|------|------|
63+
| `src/main/java/com/hubspot/jinjava/Jinjava.java` | Entry point; `copyGlobalContext()` runs on every render |
64+
| `src/main/java/com/hubspot/jinjava/interpret/JinjavaInterpreter.java` | Main render loop; walks the AST |
65+
| `src/main/java/com/hubspot/jinjava/interpret/Context.java` | Scope management; library/tag/filter registration |
66+
| `src/main/java/com/hubspot/jinjava/tree/TreeParser.java` | Tokenization + AST construction |
67+
| `src/main/java/com/hubspot/jinjava/el/ExpressionResolver.java` | JUEL expression evaluation |
68+
| `src/main/java/com/hubspot/jinjava/tree/output/OutputList.java` | Output assembly |
69+
| `src/main/java/com/hubspot/jinjava/lib/tag/ForTag.java` | For-loop tag (dominates template benchmarks) |
70+
| `src/main/java/com/hubspot/jinjava/util/ScopeMap.java` | Scoped variable lookup |
71+
72+
## Files in Scope
73+
74+
All files under `src/main/java/com/hubspot/jinjava/` are fair game for optimization.
75+
76+
## Off Limits
77+
78+
Do NOT modify:
79+
- `src/test/` — test code and benchmark harness must remain unchanged
80+
- `pom.xml` — no new dependencies
81+
- `src/test/resources/benchmarks/` — benchmark templates are fixed
82+
- Any file that would change Jinjava's public API or output behavior
83+
84+
## Constraints
85+
86+
1. **All tests must pass** — `mvn test` is the correctness gate
87+
2. **No new dependencies** — optimize with what's available
88+
3. **Backward-compatible** — no changes to public API contracts
89+
4. **Incremental** — make one optimization at a time, measure, commit if positive
90+
5. **No weakening security** — the jinjava-3.0 sandbox/validation must remain intact
91+
92+
## Baseline (jinjava-3.0, this branch)
93+
94+
```
95+
Benchmark Mode Cnt Score Error Units
96+
JinjavaBenchmark.complexTemplateBenchmark thrpt 5 5334 ± 840 ops/s
97+
JinjavaBenchmark.filterChainBenchmark thrpt 5 30969 ± 16173 ops/s
98+
JinjavaBenchmark.jinjavaCreationBenchmark thrpt 5 63083 ± 31002 ops/s
99+
JinjavaBenchmark.jinjavaWithConfigCreationBenchmark thrpt 5 73943 ± 18929 ops/s
100+
JinjavaBenchmark.simpleExpressionBenchmark thrpt 5 37724 ± 4073 ops/s
101+
JinjavaBenchmark.simpleTemplateBenchmark thrpt 5 17001 ± 444 ops/s
102+
```
103+
104+
### 2.8.x Reference (target to recover toward)
105+
106+
```
107+
Benchmark Mode Cnt Score Error Units
108+
JinjavaBenchmark.complexTemplateBenchmark thrpt 5 7703 ± 718 ops/s
109+
JinjavaBenchmark.filterChainBenchmark thrpt 5 30737 ± 1462 ops/s
110+
JinjavaBenchmark.jinjavaCreationBenchmark thrpt 5 74097 ± 15463 ops/s
111+
JinjavaBenchmark.jinjavaWithConfigCreationBenchmark thrpt 5 80160 ± 5558 ops/s
112+
JinjavaBenchmark.simpleExpressionBenchmark thrpt 5 33940 ± 2307 ops/s
113+
JinjavaBenchmark.simpleTemplateBenchmark thrpt 5 18487 ± 2051 ops/s
114+
```
115+
116+
### Delta (jinjava-3.0 vs 2.8.x)
117+
118+
| Benchmark | 2.8.x | 3.0 | Delta |
119+
|-----------|-------|-----|-------|
120+
| complexTemplateBenchmark | 7,703 | 5,334 | **-30.8%** |
121+
| simpleTemplateBenchmark | 18,487 | 17,001 | **-8.0%** |
122+
| simpleExpressionBenchmark | 33,940 | 37,724 | +11.1% |
123+
| filterChainBenchmark | 30,737 | 30,969 | +0.8% |
124+
| jinjavaCreationBenchmark | 74,097 | 63,083 | -14.9% |
125+
| jinjavaWithConfigCreationBenchmark | 80,160 | 73,943 | -7.8% |
126+
127+
## Optimization Ideas (Starting Points)
128+
129+
1. **`copyGlobalContext()` is O(n) per render** — It iterates all tags, filters, functions, and expTests on every single render call. Consider lazy copying, copy-on-write, or sharing the read-only global registrations via an immutable snapshot.
130+
131+
2. **`Context` scope chain** — `ScopeMap` lookups walk the parent chain. For deeply nested for-loops, this becomes O(depth) per variable access. Consider flattening or caching.
132+
133+
3. **`TreeParser` re-parses every time** — If the same template string is rendered multiple times (common in benchmarks and production), caching the parsed AST (Node tree) keyed by template string would skip tokenization entirely.
134+
135+
4. **`ForTag` per-iteration overhead** — Each loop iteration creates a new child scope. If scope creation is heavyweight (copying maps), consider lightweight scope push/pop.
136+
137+
5. **`ExpressionResolver` / JUEL overhead** — JUEL expression compilation may be repeated. Check if expressions can be cached after first compilation.
138+
139+
6. **String concatenation in `OutputList`** — If `OutputList.getValue()` uses `StringBuilder` inefficiently or creates intermediate strings, pre-sizing or streaming could help.
140+
141+
7. **Security validation hot path** — The 3.0 sandbox likely adds per-method-call or per-class-access validation. Profile to see if validation results can be cached (e.g., "is this method allowed?" → memoize).
142+
143+
## Progress Log
144+
145+
Use this template for each optimization attempt:
146+
147+
```
148+
### Attempt N: <short description>
149+
- **Hypothesis**: ...
150+
- **Change**: ...
151+
- **Result**: complexTemplate=X ops/s (±Y%), simpleTemplate=X ops/s (±Y%)
152+
- **Verdict**: KEEP / REVERT
153+
```

‎auto/autoresearch.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5+
PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
6+
cd "$PROJECT_DIR"
7+
8+
echo "=== Step 1: Running mvn test (correctness gate) ==="
9+
mvn test -q
10+
echo "Tests passed."
11+
12+
echo ""
13+
echo "=== Step 2: Compiling test classes ==="
14+
mvn test-compile -q
15+
16+
echo ""
17+
echo "=== Step 3: Building classpath ==="
18+
CP_FILE=$(mktemp)
19+
mvn dependency:build-classpath -Dmdep.includeScope=test -Dmdep.outputFile="$CP_FILE" -q
20+
21+
echo ""
22+
echo "=== Step 4: Running JMH benchmarks ==="
23+
java -cp "$(cat "$CP_FILE"):target/classes:target/test-classes" \
24+
com.hubspot.jinjava.benchmarks.JinjavaBenchmark
25+
26+
rm -f "$CP_FILE"
27+
echo ""
28+
echo "=== Done ==="

0 commit comments

Comments
 (0)