About DeepOBS
DeepOBS is a benchmarking suite that drastically simplifies, automates and improves the evaluation of deep learning optimizers. It can evaluate the performance of new optimizers on a variety of real-world test problems and automatically compare them with realistic baselines.
Fast
You only have to code your new optimizer, we take care of the benchmarking for you!
Fair
Automatically use competitive baselines!
Comparable
The same test problems for all optimizers!
How to Install DeepOBS
You can install the latest stable release of DeepOBS using pip
pip install deepobs You are ready to dive into the tutorials found in the DeepOBS documentation.
Leaderboard
Overview over the current optimizer leaderboard on the DeepOBS test problems. Click on See Full Results to see the plots and tables of the full benchmarking results.
Quadratic Deep
A 100-dimensional noisy quadratic problem with an eigenspectrum similar to the one reported for deep neural networks.
| Optimizer | Test Loss | Speed | |
|---|---|---|---|
| #1 | Momentum | 87.05 | 70.5 |
| #2 | Adam | 87.11 | 39.9 |
| #3 | SGD | 87.40 | 51.1 |
MNIST - VAE
A basic variational autoencoder for the MNIST data set with three convolutional and three deconvolutional layers.
| Optimizer | Test Loss | Speed | |
|---|---|---|---|
| #1 | Adam | 27.83 | 1.0 |
| #2 | SGD | 38.46 | 1.0 |
| #3 | Momentum | 52.93 | 1.0 |
F-MNIST - CNN
A simple convolutional network for the Fashion-MNIST data set, consisting of two conv and two fully-connected layers.
| Optimizer | Test Accuracy | Speed | |
|---|---|---|---|
| #1 | Adam | 92.34 % | 40.1 |
| #2 | SGD | 92.27 % | 40.6 |
| #3 | Momentum | 92.14 % | 59.1 |
CIFAR-10 - CNN
A slightly larger convolutional network for the Cifar-10 data set, with three conv and three fully-connected layers.
| Optimizer | Test Accuracy | Speed | |
|---|---|---|---|
| #1 | Adam | 84.75 % | 36.0 |
| #2 | Momentum | 84.41 % | 40.7 |
| #3 | SGD | 83.71 % | 42.5 |
F-MNIST - VAE
A basic variational autoencoder for the Fashion-MNIST data set with three convolutional and three deconvolutional layers.
| Optimizer | Test Loss | Speed | |
|---|---|---|---|
| #1 | Adam | 23.07 | 1.0 |
| #2 | SGD | 23.80 | 1.0 |
| #3 | Momentum | 59.23 | 1.0 |
CIFAR-100 - All CNN C
Variant C of the All Convolutional Network from Striving for Simplicity for the CIFAR-100 data set consisting solely of convolutional layers.
| Optimizer | Test Accuracy | Speed | |
|---|---|---|---|
| #1 | Momentum | 60.33 % | 72.8 |
| #2 | SGD | 57.06 % | 128.7 |
| #3 | Adam | 56.15 % | 152.6 |
SVHN - Wide ResNet 16-4
The Wide ResNet 16-4 for the Street View House Numbers data set using the variant with 16 conv layers and a widening factor of 4.
| Optimizer | Test Accuracy | Speed | |
|---|---|---|---|
| #1 | Momentum | 95.53 % | 10.8 |
| #2 | SGD | 95.37 % | 28.3 |
| #3 | Adam | 95.25 % | 12.1 |
Tolstoi - Char RNN
A recurrent neural network for character-level language modeling on the novel War and Peace by Leo Tolstoy using two LSTM layers.
| Optimizer | Test Accuracy | Speed | |
|---|---|---|---|
| #1 | SGD | 62.07 % | 47.7 |
| #2 | Momentum | 61.30 % | 88.0 |
| #3 | Adam | 61.23 % | 62.8 |
DeepOBS is Open Source
If you use DeepOBS in your research, please cite our paper.
@InProceedings{ schneider2018deepobs,
Title = {Deep{OBS}: A Deep Learning Optimizer Benchmark Suite},
Author = {Frank Schneider and Lukas Balles and Philipp Hennig},
Booktitle = {International Conference on Learning Representations},
Year = {2019},
Url = {https://openreview.net/forum?id=rJg6ssC5Y7}
}