Lecture 1: History and Biological Inspiration

Learning Objectives

After completing this lecture, you should be able to:

1. Explain the historical development of neural networks from 1943-1969
2. Describe how biological neurons process information
3. Identify the key contributions of McCulloch, Pitts, Hebb, and Rosenblatt
4. Understand the analogy between neural networks and decision-making committees
5. Recognize why the brain inspired early AI researchers

Key Concepts

1. The Investment Committee Analogy

Before diving into mathematics, consider how an investment committee makes decisions:

The Process:

1. Each analyst provides input based on their expertise
2. Opinions are weighted by seniority/experience
3. Weighted votes are summed
4. If the total exceeds a threshold: Buy

With a threshold of 0, the decision is: Don’t Buy

This is exactly how a perceptron works! The artificial neuron:

• Receives inputs (analyst opinions)
• Applies weights (seniority)
• Sums the weighted inputs
• Applies a threshold decision rule

2. Historical Timeline: 1943-1969

1943: McCulloch-Pitts Neuron

Warren McCulloch (neuroscientist) and Walter Pitts (mathematician) published “A Logical Calculus of Ideas Immanent in Nervous Activity.”

Key Insight: Neurons can be modeled as logical gates that perform computation.

Their model:

• Binary inputs (0 or 1)
• Binary output (fires or doesn’t fire)
• Threshold-based activation

Limitation: Weights were fixed, not learned.

Click chart to view Python source code

1949: Hebbian Learning

Donald Hebb proposed how neural connections strengthen:

“Neurons that fire together, wire together.”

Hebb’s Rule: When neuron A consistently activates neuron B, the connection between them strengthens.

Mathematically:

delta_w = learning_rate * x * y

Where:

• delta_w = change in weight
• x = input activation
• y = output activation

This was the first learning rule for neural networks!

1958: The Perceptron

Frank Rosenblatt at Cornell built the Mark I Perceptron, a physical machine that could:

• Learn to classify visual patterns
• Adjust weights automatically through training

The New York Times (1958): “The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.”

The hype was enormous - perhaps too enormous.

1969: The AI Winter Begins

Marvin Minsky and Seymour Papert published “Perceptrons,” mathematically proving that:

1. Single perceptrons cannot solve the XOR problem
2. Many interesting problems are not linearly separable

This criticism, combined with overhyped expectations, led to the first “AI Winter” - a period of reduced funding and interest in neural networks.

3. Biological Neurons

To understand artificial neurons, we must first understand their biological inspiration.

Structure of a Biological Neuron

How Neurons Communicate

1. Input: Dendrites receive chemical signals (neurotransmitters)
2. Integration: Cell body sums incoming signals
3. Threshold: If sum exceeds threshold, neuron “fires”
4. Output: Electrical signal travels down axon
5. Transmission: Signal converted to chemical at synapse

Key Numbers

• Human brain: ~86 billion neurons
• Each neuron: ~7,000 synaptic connections
• Total synapses: ~100 trillion

Click chart to view Python source code

4. From Biology to Mathematics

The artificial neuron is a simplified mathematical model:

Important: Artificial neurons are inspired by biology but do NOT accurately model real neurons. The brain is far more complex than our models suggest.

Key Formulas

McCulloch-Pitts Neuron

y = 1 if sum(x_i) >= threshold
y = 0 otherwise

Hebbian Learning Update

w_new = w_old + eta * x * y

Where eta is the learning rate.

Finance Application: Stock Screening

A perceptron can act as a simple stock screener:

Inputs (features):

• P/E Ratio
• 6-month momentum
• Trading volume
• Debt-to-equity ratio
• Earnings surprise

Output:

• 1 = Add to portfolio (Buy)
• 0 = Pass

The key insight: Instead of manually setting decision rules, the perceptron learns appropriate weights from historical winners and losers.

Practice Questions

Conceptual Understanding

Q1: Why did McCulloch and Pitts model neurons as binary threshold units?

Q2: What is Hebb’s learning rule in simple terms?

Q3: Why was the XOR problem significant for perceptrons?

Q4: True or False: Artificial neurons accurately simulate biological neurons.

Q5: In the investment committee analogy, what corresponds to the “weights” in a perceptron?

Application

Q6: A loan approval perceptron has these inputs: income, credit score, employment years. What might negative weights indicate for certain features?

Q7: Why did the AI winter occur after Minsky and Papert’s book?

Reading List

Essential Reading

• McCulloch & Pitts (1943) - “A Logical Calculus of Ideas Immanent in Nervous Activity” - The foundational paper
• Rosenblatt (1958) - “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain”

Historical Context

• Minsky & Papert (1969) - “Perceptrons” - The book that started the AI winter
• Olazaran (1996) - “A Sociological Study of the Official History of the Perceptrons Controversy”

Accessible Introductions

• Nielsen, Chapter 1 - “Using neural nets to recognize handwritten digits” (online)
• 3Blue1Brown - “But what is a neural network?” (YouTube)

Finance Connection

• Dixon, Halperin & Bilokon (2020) - “Machine Learning in Finance” - Chapter on neural network history

Summary

This lecture covered:

1. The investment committee analogy - Neural networks are like weighted voting systems
2. Historical development (1943-1969) - From McCulloch-Pitts to the AI winter
3. Biological inspiration - How real neurons inspired artificial ones
4. Key limitations - Why single perceptrons faced fundamental constraints

Next Lecture: Perceptron Fundamentals - We dive deep into the mathematics and learning algorithm of the perceptron.