Deep learning | Study notes
🧠 Basic equations for the fully connected Neural Network
3 min readDec 20, 2018
Forward propagation :
Cost functions :
. Quadratic
. Cross-entropy
Backward propagation :
dz2 = a2 - y
dz1 = np.dot(W2.T, dz2) * relu_prime(z1)dW2 = np.dot(dz2,a1.T) / m
db2 = np.sum(dz2, axis = 1, keepdims = True) / mdW1 = np.dot(dz1,x.T) / m
db1 = np.sum(dz1, axis = 1, keepdims = True) / m
Parameters update :
W2 = W2 - learning_rate * dW2
b2 = b2 - learning_rate * db2W1 = W1 - learning_rate * dW1
b1 = b1 - learning_rate * db1
L2 Regularisation :
Feature normalisation :
Weights initialisation preventing vanishing gradients :
# if activation is relu :
W1 = np.random.randn(n_l1, n_l0) * np.sqrt(2 / n_l0)# if activation is tanh :
W1 = np.random.randn(n_l1, n_l0) * np.sqrt(1 / n_l0)
Learning rate decay :
# for i in range(num_epochs)
alpha = 1 / (1 + decay_rate * i) * alpha_0Precision, Recall, F1 Score :
Exponentially weighted average :
Gradient descent with momentum :
RMSProp :
Adam :
Batch normalisation :
Activation functions :
. ReLu
. ReLu’
. Tanh
. Tanh’
. Sigmoid
. Sigmoid’
. Softmax
. Softmax’
