Lecture 4: Activation and Loss Functions

Learning Objectives

After completing this lecture, you should be able to:

1. Explain why activation functions are necessary
2. Compare sigmoid, tanh, and ReLU activation functions
3. Choose appropriate activation functions for different layers
4. Understand the purpose of loss functions
5. Apply MSE and cross-entropy loss to appropriate problems
6. Connect loss functions to the optimization process

Key Concepts

1. Why Activation Functions?

Recall from Lecture 3: Without non-linear activation functions, any multi-layer network collapses to a single linear transformation.

Activation functions provide:

• Non-linearity (essential for learning complex patterns)
• Bounded output (for some functions)
• Differentiability (needed for gradient-based learning)

2. The Sigmoid Function

The sigmoid (logistic) function was historically the most popular activation.

Formula:

sigmoid(z) = 1 / (1 + e^(-z))

Properties:

• Output range: (0, 1)
• Smooth, differentiable everywhere
• Centered at 0.5

Derivative:

sigmoid'(z) = sigmoid(z) * (1 - sigmoid(z))

Advantages:

• Outputs interpretable as probabilities
• Smooth gradient

Disadvantages:

• Vanishing gradient for large

  z

• Not zero-centered
• Computationally expensive (exponentials)

Click chart to view Python source code

3. The Tanh Function

Tanh is a scaled and shifted sigmoid.

Formula:

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Or equivalently:

tanh(z) = 2 * sigmoid(2z) - 1

Properties:

• Output range: (-1, 1)
• Zero-centered
• Smooth, differentiable everywhere

Derivative:

tanh'(z) = 1 - tanh^2(z)

Advantages:

• Zero-centered (better gradient flow)
• Stronger gradients than sigmoid

Disadvantages:

• Still suffers from vanishing gradients
• Computationally expensive

Click chart to view Python source code

4. The ReLU Function

ReLU (Rectified Linear Unit) is the most popular modern activation.

Formula:

ReLU(z) = max(0, z)

Properties:

• Output range: [0, infinity)
• Not bounded above
• Not differentiable at z = 0 (use subgradient)

Derivative:

ReLU'(z) = { 1 if z > 0
           { 0 if z < 0
           { undefined at z = 0 (typically use 0 or 1)

Advantages:

• No vanishing gradient for positive values
• Computationally efficient (no exponentials)
• Sparse activation (many zeros)

Disadvantages:

• “Dead ReLU” problem: neurons can get stuck at 0
• Not zero-centered
• Unbounded (can cause exploding values)

Click chart to view Python source code

5. Activation Function Comparison

Click chart to view Python source code

Modern best practice:

• Hidden layers: ReLU (or variants like Leaky ReLU, ELU)
• Output layer: Depends on task (see below)

6. Choosing Output Activation

The output layer activation depends on the problem type:

7. The Universal Approximation Theorem

Theorem (Cybenko, 1989; Hornik, 1991): A feedforward network with a single hidden layer containing a finite number of neurons can approximate any continuous function on compact subsets of R^n, given appropriate activation functions and sufficient neurons.

Implications:

• Neural networks are theoretically capable of learning any pattern
• BUT: The theorem doesn’t tell us how many neurons are needed
• AND: It doesn’t tell us how to find the right weights

Practical reality:

• Deeper networks often work better than very wide shallow ones
• Finding good weights requires proper training algorithms

8. Loss Functions: Measuring Mistakes

A loss function quantifies how wrong the network’s predictions are.

Purpose:

• Provides a single number to minimize
• Guides the optimization process
• Different losses for different problems

Notation:

• y_true (or y): The correct answer
• y_pred (or y-hat): The network’s prediction
• L: The loss value

9. Mean Squared Error (MSE)

MSE is the standard loss for regression problems.

Formula:

MSE = (1/n) * sum((y_true - y_pred)^2)

For a single example:

L = (y_true - y_pred)^2

Properties:

• Always non-negative
• Zero only when predictions are perfect
• Heavily penalizes large errors (quadratic)

Derivative:

dL/dy_pred = -2 * (y_true - y_pred)

Use for:

• Stock price prediction
• Portfolio return forecasting
• Any continuous target variable

Click chart to view Python source code

10. Binary Cross-Entropy Loss

Cross-entropy is the standard loss for classification problems.

Formula (for binary classification):

BCE = -[y_true * log(y_pred) + (1 - y_true) * log(1 - y_pred)]

Intuition:

• When y_true = 1: Loss = -log(y_pred) -> want y_pred close to 1
• When y_true = 0: Loss = -log(1 - y_pred) -> want y_pred close to 0

Properties:

• Always non-negative
• Penalizes confident wrong predictions heavily
• Works well with sigmoid output

Derivative:

dL/dy_pred = -y_true/y_pred + (1-y_true)/(1-y_pred)

Use for:

• Buy/sell classification
• Fraud detection
• Any binary outcome

Click chart to view Python source code

11. Choosing the Right Loss Function

Finance examples:

• Predicting next-day return: MSE + Linear output
• Predicting up/down: Cross-entropy + Sigmoid output
• Predicting sector (11 classes): Categorical cross-entropy + Softmax output

Key Formulas

Activation Functions

Loss Functions

Mean Squared Error:

L_MSE = (1/n) * sum_{i=1}^{n} (y_i - y_hat_i)^2

Binary Cross-Entropy:

L_BCE = -(1/n) * sum_{i=1}^{n} [y_i*log(y_hat_i) + (1-y_i)*log(1-y_hat_i)]

Finance Application: Output Design

When building financial prediction models, output design matters:

Stock Direction Prediction:

• Output: Sigmoid (probability of going up)
• Loss: Binary cross-entropy
• Interpretation: P(up) = 0.7 means 70% confidence in upward movement

Return Prediction:

• Output: Linear (unbounded)
• Loss: MSE
• Interpretation: Predicted return of 0.02 means expected 2% return

Risk Level Classification (Low/Medium/High):

• Output: Softmax (3 neurons)
• Loss: Categorical cross-entropy
• Interpretation: [0.1, 0.3, 0.6] means 60% probability of high risk

Practice Questions

Mathematical Understanding

Q1: Calculate sigmoid(0), sigmoid(2), and sigmoid(-2).

Q2: Given y_true = 1 and y_pred = 0.9, calculate the binary cross-entropy loss.

Q3: For the same prediction (y_pred = 0.9), what would the BCE loss be if y_true = 0?

Conceptual Understanding

Q4: Why does ReLU help with the vanishing gradient problem?

Q5: What is the “dead ReLU” problem and how might it be addressed?

Q6: Why is cross-entropy preferred over MSE for classification?

Application

Q7: You’re building a model to predict stock volatility (always positive). What output activation and loss would you use?

Reading List

Essential Reading

• Nielsen, Chapter 3 - “Improving the way neural networks learn” (online)
• Goodfellow et al., Chapter 6 - Deep Learning - “Deep Feedforward Networks”

Activation Functions

• Nair & Hinton (2010) - “Rectified Linear Units Improve Restricted Boltzmann Machines”
• Glorot et al. (2011) - “Deep Sparse Rectifier Neural Networks”

Theoretical Foundation

• Cybenko (1989) - “Approximation by Superpositions of a Sigmoidal Function”
• Hornik (1991) - “Approximation Capabilities of Multilayer Feedforward Networks”

Finance Applications

• Heaton et al. (2016) - “Deep Learning for Finance” - Discusses output design for financial problems

Summary

This lecture covered:

1. Why activation functions - Enable non-linearity in neural networks
2. Sigmoid - Smooth, bounded (0,1), but vanishing gradients
3. Tanh - Zero-centered, but still vanishing gradients
4. ReLU - Fast, no vanishing gradient, but dead neurons possible
5. Loss functions - MSE for regression, cross-entropy for classification
6. Output design - Match activation and loss to the problem type

Key Takeaway: The combination of activation function and loss function determines how well your network can learn and what problems it can solve.

Next Lecture: Gradient Descent and Backpropagation - We’ll learn how neural networks actually find good weights.