A Public Wall-Clock Leaderboard for Fine-Tuning Techniques

LoRA Speedrun

Open Source Project

2024

Why Fine-Tuning Efficiency Matters

01

Growing Model Sizes Modern LLMs have billions of parameters — full fine-tuning is expensive
02

Compute Costs Training time directly impacts GPU costs and iteration speed
03

Real-World Impact Faster fine-tuning means faster experimentation and deployment
04

Need for Comparison Practitioners need objective, reproducible benchmarks to choose techniques

03

The first public wall-clock time benchmark for fine-tuning methods

01

Open Source Benchmark GitHub repository with standardized testing framework
02

Wall-Clock Time Focus Measures actual real-world execution time, not theoretical FLOPS
03

Fair Comparison Same hardware, same datasets, same model architectures
04

Community Driven Anyone can submit new techniques and results
05

Transparent Results Full reproducibility with code and configuration details

Comprehensive benchmarking across multiple dimensions

timer

Actual training duration in seconds/minutes

speed

Peak GPU memory consumption during training

storage

Storage footprint of saved model weights

functions

Performance metrics on downstream tasks

bolt

Samples processed per second

compare_arrows

Steps needed to reach target performance

64

Hacker News Points

Strong community engagement on launch — practitioners are paying attention

"Wall-clock time is the metric that actually matters for practitioners —
theoretical efficiency doesn't pay the GPU bill."

The Case for Real-World Benchmarks

Explore the benchmark and contribute your results

github.com/Saivineeth147/lora-speedrun