Neural Network
A neural network is a computational model composed of interconnected layers of simple processing units called neurons, loosely inspired by biological brains. Each neuron computes a weighted sum of its inputs, applies a nonlinear activation…
Definition
A neural network is a computational model composed of interconnected layers of simple processing units called neurons, loosely inspired by biological brains. Each neuron computes a weighted sum of its inputs, applies a nonlinear activation function, and passes the result forward. By adjusting the weights through training, a neural network learns to map inputs to desired outputs.
Overview
A neural network consists of an input layer, one or more hidden layers, and an output layer. Each connection between neurons has an associated weight, and each neuron typically has a bias term. During a forward pass, data flows through the network: each neuron computes a weighted sum of its inputs plus bias, then applies a nonlinear activation function (such as ReLU, sigmoid, or tanh) that allows the network to model complex, non-linear relationships rather than just linear combinations. Training a neural network means finding the weight values that make the network's outputs match desired targets as closely as possible. This is done by defining a loss function that measures prediction error, then using backpropagation to compute how each weight contributed to that error, and gradient descent to iteratively nudge weights in the direction that reduces the loss. The simplest neural networks are 'feedforward' or 'fully connected,' where every neuron in one layer connects to every neuron in the next. Specialized architectures add structure suited to particular data types: convolutional layers for images, recurrent connections for sequences, and attention mechanisms (as in transformers) for modeling relationships across an entire sequence at once. The term 'deep learning' refers specifically to neural networks with many hidden layers. A single-layer network (perceptron) can only learn linearly separable patterns, but stacking layers with nonlinear activations gives networks the theoretical ability to approximate essentially any function, which is why deep neural networks power modern AI systems from image recognizers to large language models. Understanding neural networks is foundational to understanding virtually every modern AI technique.
Key Concepts
- Composed of layers of interconnected neurons, each applying a weighted sum plus nonlinear activation
- Learns by adjusting weights via backpropagation and gradient descent to minimize a loss function
- Nonlinear activation functions (ReLU, sigmoid, tanh) enable modeling of complex relationships
- Architecture varies by data type: fully connected, convolutional, recurrent, or attention-based
- Deeper networks (more layers) can represent more abstract, hierarchical features
- Requires large amounts of labeled or structured data and significant compute to train well
- Universal approximation theory shows sufficiently large networks can model arbitrary functions
- Forms the computational backbone of deep learning and most modern generative AI