LEARNING ADVENTURE
Changing the connections
Sometimes, having too many learning sources can be confusing, and this is especially true for deep learning. My solution? I kept diving into multiple articles, books, and videos. Along the way, I've picked up knowledge (and likely written some code) in the following areas of deep learning:
Basic: Layers (Dense, BatchNorm, Dropout, ...) - Regularization Techniques (Dropout, BatchNorm, Data augmentation, DropConnect, ...) - Loss functions (Hinge loss, Cross Entropy losses, MSE, MAE, ...) - Optimization algorithms (SGD, SGD+Momentum, Nestrov momentum, Adagrad, RMSProp, Adam) - Preprocessing
Computer Vision: Layers (Conv., TransConv., GDN, IGDN, Pooling, ...)
Recurrent Neural Networks: Layers (Simple RNN, LSTM, GRU, Conv. LSTM, Bidirectionals)
TinyML: Quantization - Pruning - Clustering - Neuron Merging - Knowledge Distillation - Early Exit Networks - NAS - ...
Generative Adversarial Networks: CGAN - DCGAN - CycleGAN - ...
Reinforcement Learning: Essentials
Network Structures: AE (AE, VAE, CAE, ...) - Classification (AlexNet, VGG, GoogLeNet, ResNe(x)t, NiN, FractalNet, DenseNet, SqueezeNet, ...) - Detection ((Fast(er)) R-CNN, SPPNet, YOLO, ...) - Segmentation (Mask R-CNN, SegNet, U-Net, ...)
Applications: Face detection and recognition - NLP and translation - Telecommunication coding - Data generation - Style transfer - Word embedding - ...
Etc.
Through this process, I've found the following sources so helpful.
CS231n (Stanford)
One of the best learning resources for me was the video series on Convolutional Neural Networks for Visual Recognition (CS231n) from Stanford University. I watched both the 2016 and 2017 course videos, and while Andrej Karpathy took center stage in the first one, I found the 2017 version even more engaging. The way they explained the concepts was so beautiful and intuitive - I absolutely loved it. Hats off to them!
Coursera
One of the famous learning sources available online is the Deep Learning Specialization instructed by Andrew NG, a pioneer in deep learning, available on Coursera.
The specialization consists of five courses, titled: Neural Networks and Deep Learning - Improving Deep Neural Networks - Structuring Machine Learning Projects - Convolutional Neural Networks - Sequence Models. These courses start from the basic elements and cover a lot of deep learning subjects in detail. With the exercises and quizzes offered in each course, you can make sure you truly grasp the material. Although each course is designed to take about four weeks to complete, thanks to my prior knowledge, I was able to finish some of them in under two days, all with a perfect score of 100/100 (including the optional parts) :)
Coursera
Introduction to Deep Learning (MIT)
MIT's 6.S191 course (Introduction to Deep Learning) is another excellent learning resource. It's concise yet highly informative, covering the essential topics in deep learning. They even go beyond the typical scope of most deep learning courses, touching on more advanced subjects like Reinforcement Learning.
Deep Learning (Book)
This book is perfect for anyone wanting to learn deep learning from the ground up, especially if you enjoy math. Just take a look at the authors: Ian Goodfellow and Yoshua Bengio (I have a bit of a weird feeling about the third author - I can't shake the sense that he might have been their typing machine :). Anyway, this book actually serves as the main reference for deep learning courses at universities. While its content is excellent, it does take quite a bit of time. I only made it halfway through Part II before life got in the way. Hopefully, I'll be able to finish it in my free time...
Deep Learning (Book)
Efficient ML (MIT)
When it comes to TinyML, who else can you name to be more influential than Song Han? MIT's 6.5940 course (Efficient ML) is a great resource for learning about the latest techniques in deep learning. The course covers a wide range of topics, including model compression, quantization, and pruning. It's a must-watch for anyone interested in making deep learning models more efficient.
