Environment Setup

conda create --name NeuralNetworksAndDeepLearning
conda activate NeuralNetworksAndDeepLearning
conda install mkl-service
conda install mkl
conda install -c anaconda ipykernel nbconvert
python -m ipykernel install --user --name=NNaDL

NNaDL = Neural Networks and Deep Learning
Switch to NNaDL kernel in Jupyter Notebook

Contents

1. Basic matrix operations in NumPy. Visualization of weights of the neural network. View

• implementation of the sigmoid activation function
• implementation of the feed forward operation in one-layer neural network
• visualization of weights of the neural network

2. Visualizations and classification based on MNIST - digits dataset. View

• visualizations of the distribution of MNIST digits
• Principal Component Analysis (PCA) - 2D and 3D (plotly)
• T-distributed Stochastic Neighbour Embedding (T-SNE)
• Classification using SVM
• Analysis of confusion matrix
• boolean indexing in NumPy

3. Restricted Boltzmann Machine and Contrastive Divergence algorithm. View

• Restricted Boltzmann Machine
• Contrastive Divergence algorithm
• Gibbs sampling
• Stochastic gradient descent

4. Evaluation and comparison of CDk and PCD algorithms. View

• Implementation of Persistent Contrastive Divergence algorithm
• Comparision of RBM's training algorithms based on MNIST dataset

5. Training RBM with Momentum and Introduction to DBNs (Deep Belief Networks). View

• RBM training with Momentum - modification of classical SGD algorithm
• DBN greedy layer-wise training
• DBN sampling

6. Backpropagation algorithm. View

• Two phases of the algorithm: forward pass and error backpropagation
• MLP training with Backpropagation and minibatched variant of SGD

7. L1L2 Regularization and initialization of MLP weights. View

• L1 and L2 regularization
• MLP pre-training using DBN
• Comparison of results of plan MLP and pre-trained MLP

8. ReLU activation function and Max-Norm Regularization. View

• Changing Sigmoid to ReLU activation function
• Max-Norm Reguralization. Introduction of competitive weights
• Initializing MLP with DBN weights

9. Dropout Regularization. View

10. Autoencoder. View

• Autoencoder initialization with DBN weights
• Autoencoder training

11. Autoencoder training with Nesterov accelerated gradient descent. View

12. Convolutional Neural Network. View

13. Negative Sampling Algorithm. View