(T) I attended last week three presentations from the Baidu AI Lab in Sunnyvale, part of the SF Big Analytics meet-up in San Francisco. Professor Andrew Ng’s Stanford University Professor and Coursera Co-Founder is leading Baidu’s research in Artificial Intelligence with a great team of scientists.
Following are my notes from the lecture presentations:
Professor Andrew Ng: Why is deep learning taking off?
An analogy to a rocket….Engine <=>large neural networks – Fuel <=>data
- 2007: CPU -> 1 million connections
- 2008: GPU -> 10 million connections
- 2011: Many CPUs = Cloud -> 1 billion connections
- 2015: Many GPUs = HPC -> 100 billion connections
Bryan Catanzaro: Why is HPC so important to AI?
Training deep neural networks is an HPC challenge
Using HPC hardware and traditional software approaches reduces training time
This lets scale to large models and data sets
Scaling brings AI progress
Many GPUs (scalable parallel processing) versus many CPUs (fast serial processing):
- HPC (many GPUs): Computing at the limit – FLOPS/memory bandwidth – tightly coupled
- Cloud (many CPUs): Hardware at the limit – I/O bound – Confederated
Awni Hannun: Deep learning for speech recognition
Audio -> acoustic model/phonemes -> prediction of words -> language model with probability of word
Model: bi-directional recurrent neural network (RNN) with CTC (Connectionist Temporal Classification)
Data: over 100,00 hours of synthesized data – (speech + noise)
Computation: GPU/ model and data parallel
- Must handle variable length input and output
- Loss function: audio -> letters
Reference: A Silicon Valley Insider, Deep Dive into Deep Learning
Note: The picture above is from the talk.
Copyright © 2005-2015 by Serge-Paul Carrasco. All rights reserved.
Contact Us: asvinsider at gmail dot com.
Categories: Artificial Intelligence, Deep Learning, Machine Learning