logoalt Hacker News

thesz • yesterday at 6:50 PM • 1 reply • view on HN

ADAM does not work on simple convex problems [1].

  [1] https://parameterfree.com/2020/12/06/neural-network-maybe-evolved-to-make-adam-the-best-optimizer/
  [2] https://arxiv.org/pdf/1905.09997
[1] refers to [2], which shows that ADAM is not as efficient as gradient descent with line search on some problems, including neural networks.

Replies

_alternator_ • yesterday at 7:41 PM

I'll point out that "does not work" is not the same as "not as efficient" :) But it does seem the Adam paper had an error.

I think that Nesterov's first order method is the most efficient general first order algorithm on convex problems, so anything else is in some sense worse. (Edit: removed incorrect ADAM comment.)

➕ show 1 reply