Skip to content

Phase 3.5 Performance Validation Report

Date: 2026-09-03
Status: ⏳ In Progress
Branch: feature/phase3.5-performance-validation

Summary

Phase 3.5 validates the QwenBatchScheduler performance against established targets: - Single-request latency: ≤1.05× baseline (no regression) - 2 concurrent throughput: ≥1.7× baseline - 4 concurrent throughput: ≥3.0× baseline - 8 concurrent throughput: ≥4.5× baseline

Test Environment

Hardware: - GPU: [TO BE FILLED] - CPU: [TO BE FILLED] - RAM: [TO BE FILLED]

Software: - CUDA: [TO BE FILLED] - Python: [TO BE FILLED] - PocketLLM: commit [TO BE FILLED]

Checkpoint: - Model: Qwen3.5 - Path: [TO BE FILLED] - Params: [TO BE FILLED]

Test 1: Single-Request Latency

Goal: Verify batch mode doesn't introduce significant overhead for single requests.

Configuration: - Prompt length: 32 tokens - Max new tokens: 128 - Warmup runs: 3 - Test runs: 10

Results

[TO BE FILLED - Output from bench_single_request_latency.py]

Analysis: - Serial mode avg latency: [TO BE FILLED] - Batch mode avg latency: [TO BE FILLED] - Ratio: [TO BE FILLED]× - Verdict: [PASS/FAIL]

Test 2: Concurrent Request Throughput

Goal: Measure throughput improvement with concurrent requests.

Configuration: - Prompt length: 32 tokens - Max new tokens: 128 - Concurrency levels: 2, 4, 8 - Warmup runs: 1 - Test runs per level: 3

Results

[TO BE FILLED - Output from bench_concurrent_throughput.py]

Analysis

Concurrency Serial (req/s) Batch (req/s) Improvement Target Status
2 [TBF] [TBF] [TBF]× ≥1.7× [TBF]
4 [TBF] [TBF] [TBF]× ≥3.0× [TBF]
8 [TBF] [TBF] [TBF]× ≥4.5× [TBF]

Overall Verdict

  • [ ] Single-request latency: PASS/FAIL
  • [ ] 2 concurrent throughput: PASS/FAIL
  • [ ] 4 concurrent throughput: PASS/FAIL
  • [ ] 8 concurrent throughput: PASS/FAIL

Final Status: [PASS/FAIL]

Performance Insights

What Went Well

[TO BE FILLED]

Bottlenecks Identified

[TO BE FILLED]

Optimization Opportunities

[TO BE FILLED]

Comparison with vLLM

[TO BE FILLED - If time permits, run same benchmark on vLLM]

Next Steps

If All Tests Pass

  1. Document results in this report
  2. Update README.md with performance numbers
  3. Create PR for Phase 3.5
  4. Merge to master
  5. Proceed to Phase 4.1 (Python unified scheduler)

If Tests Fail

  1. Profile bottlenecks with nvprof/nsight
  2. Identify root causes:
  3. Scheduler thread overhead?
  4. Prefill serialization?
  5. Slot allocation blocking?
  6. Implement fixes
  7. Re-run validation

Raw Data

All benchmark outputs saved to phase3.5_results/: - single_latency_*.txt - Single-request latency detailed output - concurrent_throughput_*.txt - Concurrent throughput detailed output - results_*.json - JSON summary of all tests

Appendix: How to Reproduce

# Run all validation tests
python scripts/run_phase3.5_validation.py /path/to/qwen3.5/checkpoint

# Or run individually
python scripts/bench_single_request_latency.py /path/to/checkpoint
python scripts/bench_concurrent_throughput.py /path/to/checkpoint

See docs/archive/phase2-phase3/phase3_5_benchmark_guide.md for detailed instructions.

Completed: [TO BE FILLED]
Next Milestone: Phase 4.1 - Python Unified Scheduler