Learning Rate Scheduler Visualizer

How to use the LR Scheduler

The learning rate schedule controls how big each update is during training, and it can make the difference between a model that converges smoothly and one that diverges or stalls. This visualizer draws the schedule on a chart so you can see exactly what your configuration does before you spend GPU hours on it.

Choose a schedule type — constant, linear decay, cosine decay, cosine with hard restarts, or step decay — and set the peak learning rate, minimum learning rate, total steps and warmup steps. The chart updates instantly, and you can enter any step number to read the precise learning rate at that point. The formulas match the common implementations in PyTorch and the Hugging Face transformers library, including linear warmup from zero to the peak value.

Typical fine-tuning settings use a short warmup followed by cosine or linear decay; for LoRA on LLMs peak rates around 1e-4 to 2e-4 are common, while full fine-tuning uses much smaller values. Use the Training Steps Calculator first to work out total and warmup steps from your dataset. The chart is drawn on a canvas in your browser; nothing is sent to a server.

About this tool

Plot learning rate schedules (constant, linear decay, cosine, cosine with restarts, step) with warmup, and read the LR at any training step. People also look for: learning rate scheduler, cosine learning rate, warmup schedule, lr schedule visualizer.

Related searches