TL;DR
- Every neuron in a neural network runs its output through an activation function. For most of AI’s early history, that function was sigmoid, and it quietly capped how deep a network could go.
- Swapping sigmoid for ReLU (Rectified Linear Unit) in 2012 is the single change most responsible for unlocking the deep learning era, AlexNet’s ImageNet win, and everything downstream of it.
- Modern LLMs and the agents built on top of them run on descendants of that swap: GELU in BERT and GPT-2/3, SwiGLU in LLaMA, Qwen, Mistral, and Gemma.
- Understanding this one “minor detail” explains why AI stalled for decades, then suddenly didn’t.
A neural network with 100 layers and a neural network with one layer can be mathematically identical.
Not similar. Identical.
Stack a hundred layers of pure matrix multiplication on top of each other, and the result collapses into a single linear equation.
All layers of the neural network will collapse into one if a linear activation function is used, because no matter the number of layers, the last layer will still be a linear function of the first, which turns the whole network into just one layer.
All that depth. All that compute. All those parameters, worth nothing more than a straight line through a scatter plot.
The thing standing between “expensive linear regression” and “the model that writes your emails” is a small nonlinear function bolted onto every single neuron. It’s called an activation function, and for about thirty years, the AI field used the wrong one.
The Function Nobody Questioned for Thirty Years
Sigmoid wasn’t a bad idea. It was a biologically reasonable one.
Real neurons fire at some rate, hit a ceiling, and can’t fire faster no matter how strong the input gets.
Neurons also cannot fire faster than a certain rate, and this biological limitation motivates the use of sigmoid activation functions, whose output range is confined to a finite interval.
Squash everything into a smooth curve between 0 and 1, mimic the brain, move on.
For shallow networks, that logic held up fine.
The trouble showed up during training, in a step called gradient descent – the process of nudging every weight in the network based on how wrong its last prediction was, layer by layer, backward. To do that nudging, the network needs the derivative (the rate of change) of the activation function at every point.
Here’s where sigmoid quietly sabotaged everything stacked behind it.
Sigmoid tends to saturate for very large positive or negative inputs, leading to the vanishing gradient problem, especially in deep networks.
Push a value far enough into sigmoid’s flat tails, and its slope approaches zero. Chain twenty of those near-zero slopes together during backpropagation, and the earliest layers of the network receive a training signal so faint it might as well be silence.
You could build a deep network. You just couldn’t teach it anything.
This is why, deep into the 2000s, most practical neural networks stayed shallow. Not because researchers lacked ambition. Because the function inside every neuron had a mathematical ceiling nobody had found a way around.
[IMAGE: Side-by-side graphs of sigmoid and ReLU functions with their derivatives, showing sigmoid’s derivative flattening near zero at the tails versus ReLU’s constant derivative of 1 for positive inputs]2012: The Year the Ceiling Cracked
AlexNet didn’t reinvent the neural network. It reinvented what was inside each one.
Krizhevsky, Sutskever, and Hinton took the same convolutional architecture researchers had been experimenting with for years and replaced sigmoid and tanh with ReLU, a function so simple it looks almost like a trick question: output the input if it’s positive, output zero if it isn’t.
The result at the 2012 ImageNet competition wasn’t incremental.
AlexNet achieved a top-5 error rate of 15.3%, significantly outperforming the second-place model, which had a top-5 error rate of 26.2%.
An 11-point gap in a competition where progress had historically been measured in fractions of a percent.
The mechanism behind that jump was exactly the derivative problem sigmoid never solved.
This simple function, f(x) = max(0, x), solved the vanishing gradient problem and trained six times faster than comparable networks with tanh units.
Positive inputs get a derivative of exactly 1, no matter how deep the layer. No saturation. No fading signal. A deep neural network could finally learn using its earliest layers as effectively as its last ones.
Pause on that for a second, because it’s easy to read past.
The architectures that make GPT, image classifiers, and speech models possible today aren’t fundamentally more clever than what existed in the 1990s. They’re deeper. And they’re only deep because one function inside each neuron stopped erasing its own training signal on the way backward.
That’s the whole story of why deep learning took twenty extra years to arrive. Not missing data. Not missing compute, though both mattered. A tiny function nobody had swapped out yet.
ReLU Wasn’t Perfect Either
The fix introduced its own failure mode, and it’s worth knowing about, because it shows activation functions aren’t a solved problem, just a continuously improving one.
Feed a ReLU neuron enough negative inputs, and its output sits at zero forever. Zero output means zero gradient, which means zero learning.
There is also a problem called Dying ReLU which could make the activations zero for all inputs, and this problem is being mitigated by a modified version of ReLU called Leaky ReLU, which allows small gradient values for negative inputs.
A neuron can die mid-training and never come back. In a large enough network, some fraction of neurons doing this is tolerable. It’s still a real cost, and it’s why the story of activation functions didn’t end at ReLU in 2012, it just accelerated.
What This Means for the LLM Sitting Behind Every Agent You Use
The original Transformer architecture, the one underneath GPT and everything that followed, didn’t reinvent the activation function debate. It inherited it.
The initial activation function in the transformer block is rectified linear unit (ReLU).
It didn’t stay that way.
BERT, GPT-2, GPT-3, ViT and most early Transformers used GELU in their feed-forward blocks – a smoother, probabilistic cousin of ReLU that weights inputs by how likely they are to matter rather than hard-cutting at zero.
Then the field moved again.
Shazeer (2020) showed that activation functions with gated linear units (GLU) improve transformer training, and Scao et al. (2022) demonstrated the SwiGLU activation function outperforms the GELU activation function.
SwiGLU adds a second, learned “gate” that decides how much of each signal to let through, instead of applying the same fixed curve to everything.
That gated function is running inside the model behind your last AI conversation right now.
SwiGLU, which computes SiLU applied to one projection and multiplies it by a second projection, has since become the default activation in LLaMA, Qwen-2.5, Mistral, and Gemma.
The pattern hasn’t changed since 2012. Someone finds a slightly better nonlinear function, swaps it in, and the ceiling on what’s trainable moves up again. Every reasoning step an AI agent takes, every plan it revises, every tool call it decides to make, runs through millions of these gated activations doing exactly this kind of decision-making at the level of a single number.
[IMAGE: Timeline showing the progression of dominant activation functions from Sigmoid (1990s) to ReLU (2012, AlexNet) to GELU (2018, BERT/GPT-2) to SwiGLU (2020-2023, LLaMA and successors), with a one-line note on what problem each one solved]The Detail Was Never Minor
It’s tempting to file activation functions under “implementation detail,” something for the research team to tune and everyone else to ignore. The history says otherwise.
A function so small it fits in one line of code determined whether deep learning worked at all for the better part of three decades. It’s still determining, right now, which architectures scale cleanly to hundreds of billions of parameters and which ones hit a wall.
If you’re evaluating AI vendors, architecting agent infrastructure, or just trying to understand why this technology arrived when it did rather than in 1995, the activation function is where that answer actually lives, not in the marketing copy about “next-generation AI.” The next time someone describes a model’s size in parameters or its context window in tokens, ask what’s happening inside each neuron. That’s usually the more interesting number.
Frequently Asked Questions
Why do neural networks need activation functions?
Because without one, a neural network of any depth behaves exactly like a single-layer linear model.
All layers of the neural network will collapse into one if a linear activation function is used.
The activation function is what lets stacked layers learn curved, complex decision boundaries instead of just a straight line through the data.
What happens if a neural network has no activation function, or uses a linear one?
The network loses its ability to model anything beyond a linear relationship, regardless of how many layers it has.
No matter the number of layers in the neural network, the last layer will still be a linear function of the first layer, so a linear activation function turns the neural network into just one layer.
What is the vanishing gradient problem?
It’s what happens when an activation function’s derivative shrinks toward zero for large or extreme input values, so the training signal fades out before it reaches the earliest layers of a deep network.
Sigmoid tends to saturate for very large positive or negative inputs, leading to the vanishing gradient problem, especially in deep networks.
Deep networks trained with sigmoid or tanh often stopped learning in their earliest layers for exactly this reason.
Why is ReLU preferred over sigmoid in deep learning?
ReLU keeps a constant derivative for all positive inputs, so gradients don’t shrink as they flow backward through many layers.
ReLU helps reduce this problem since its derivative is either 0 or 1.
That, combined with being cheaper to compute than sigmoid’s exponential math, is why it became the default for hidden layers in deep networks.
What is the dying ReLU problem?
It’s the failure mode where a ReLU neuron gets stuck outputting zero for every input and stops learning entirely.
There is also a problem called Dying ReLU which could make the activations zero for all inputs.
Leaky ReLU and other variants were designed specifically to patch this by letting a small gradient through for negative inputs.
What activation function do modern LLMs like GPT and LLaMA use?
Most modern large language models have moved past plain ReLU to smoother or gated alternatives.
BERT, GPT-2, GPT-3, ViT and most early Transformers used GELU in their feed-forward blocks, and decoder-only LLMs from 2023 onward have largely moved to SwiGLU.
SwiGLU is now the default in LLaMA, Qwen-2.5, Mistral, and Gemma.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
