Calculator inputs
| Optimizer | w₀ | w* | a | α | β₁ | β₂ | Steps | Typical use |
|---|---|---|---|---|---|---|---|---|
| SGD | 4.5 | 0 | 1.8 | 0.04 | 0.00 | 0.00 | 50 | Baseline descent on smooth problems |
| Momentum | 4.5 | 0 | 1.8 | 0.05 | 0.90 | 0.00 | 50 | Faster progress across shallow valleys |
| RMSProp | 4.5 | 0 | 1.8 | 0.03 | 0.00 | 0.99 | 50 | Adaptive scaling for noisy gradients |
| Adam | 4.5 | 0 | 1.8 | 0.08 | 0.90 | 0.999 | 50 | General-purpose adaptive training setup |
Formula used
Objective: L(w) = 0.5 × a × (w − w*)² + 0.5 × λ × w²
Gradient: g = a × (w − w*) + λ × w + noise
SGD: wₜ₊₁ = wₜ − αₜgₜ
Momentum: vₜ = β₁vₜ₋₁ + αₜgₜ, wₜ₊₁ = wₜ − vₜ
Nesterov: evaluate gradient at the lookahead point wₜ − β₁vₜ₋₁
RMSProp: sₜ = β₂sₜ₋₁ + (1 − β₂)gₜ², update = αₜgₜ / (√sₜ + ε)
Adam: uses first and second moment estimates with bias correction before applying the adaptive update.
Learning-rate decay follows αₜ = α / (1 + decay × (t − 1)). Gradient clipping caps the gradient magnitude before the optimizer applies its rule.
How to use this calculator
- Choose an optimizer that matches your experiment.
- Enter the starting parameter, target value, and curvature.
- Set the learning rate and total number of steps.
- Adjust β values, epsilon, weight decay, and clipping.
- Add noise or decay if you want a rougher trajectory.
- Press the calculate button to view summary metrics and the chart.
- Download CSV for the full trajectory or PDF for a report snapshot.
Frequently asked questions
1. What does this calculator optimize?
It optimizes a single-parameter quadratic objective. That makes optimizer behavior easy to inspect mathematically while still showing learning rate, momentum, adaptive scaling, decay, clipping, and noise effects clearly.
2. Why use a one-parameter model?
A scalar parameter keeps the updates transparent. You can see how each optimizer changes direction, size, and stability without hiding the mechanics inside large matrix calculations.
3. What does curvature represent?
Curvature controls how sharply loss grows as the parameter moves away from the target. Higher curvature produces larger gradients for the same distance and can require smaller learning rates.
4. When should I increase momentum or β₁?
Increase it when progress is too slow across smooth valleys. If oscillation becomes stronger or overshooting appears, reduce the learning rate first, then consider lowering momentum slightly.
5. What is the role of β₂?
β₂ controls how strongly recent squared gradients influence adaptive scaling. Larger values smooth noisy gradients more, but adaptation becomes slower after sudden changes in gradient magnitude.
6. Why would I use gradient clipping?
Clipping limits very large gradients before the update is applied. It helps stabilize aggressive settings, especially when noise, curvature, or starting distance can produce jumps that derail convergence.
7. What does the convergence step mean?
It is the first step where the parameter gets within the tolerance distance from the target. If it shows “Not reached,” the simulation never entered that tolerance band.
8. Which optimizer is usually best?
There is no universal winner. Adam often works well quickly, Momentum can converge smoothly on stable surfaces, and RMSProp helps with noisy gradients. The best choice depends on curvature and tuning.