Primary Source: Information Processing
This is a recent Yann LeCun talk at CMU. Toward the end he discusses recent breakthroughs using GANs (Generative Adversarial Networks, see also Ian Goodfellow here and here).
LeCun tells an anecdote about the discovery of backpropagation. The first implementation of the algorithm didn’t work, probably because of a bug in the program. But they convinced themselves that the reason for the failure was that the network could easily get caught in a local minimum which prevents further improvement. It turns out that this is very improbable in high dimensional spaces, which is part of the reason behind the great success of deep learning. As I wrote here:
In the limit of high dimensionality a critical point is overwhelmingly likely to be a saddlepoint (have at least one negative eigenvalue). This means that even though the surface is not strictly convex the optimization is tractable.
This (free version!) new textbook on deep learning by Goodfellow, Bengio, and Courville looks very good. See also Michael Nielsen’s book.
If I were a young person I would be working in this area (perhaps with an eye toward applications in genomics, or perhaps working directly on the big problem of AGI). I hope after I retire they will let me hang out at one of the good AI places like Google Brain or Deep Mind :-)
The following two tabs change content below.
Stephen Hsu is vice president for Research and Graduate Studies at Michigan State University. He also serves as scientific adviser to BGI (formerly Beijing Genomics Institute) and as a member of its Cognitive Genomics Lab. Hsu’s primary work has been in applications of quantum field theory, particularly to problems in quantum chromodynamics, dark energy, black holes, entropy bounds, and particle physics beyond the standard model. He has also made contributions to genomics and bioinformatics, the theory of modern finance, and in encryption and information security. Founder of two Silicon Valley companies—SafeWeb, a pioneer in SSL VPN (Secure Sockets Layer Virtual Private Networks) appliances, which was acquired by Symantec in 2003, and Robot Genius Inc., which developed anti-malware technologies—Hsu has given invited research seminars and colloquia at leading research universities and laboratories around the world.