What is IPsec and How Does It Secure IP Traffic?
Learn what IPsec is, AH vs ESP, transport vs tunnel mode, and how IKE negotiates VPN security — with interview Q&A.
Expected Interview Answer
IPsec (Internet Protocol Security) is a suite of protocols that authenticates and encrypts IP packets at the network layer, most commonly used to build VPN tunnels between sites or remote hosts by wrapping traffic in AH (authentication) or ESP (encryption plus authentication) headers negotiated via the IKE key-exchange protocol.
IPsec operates at Layer 3, which means it secures all traffic between two endpoints regardless of the application or transport protocol above it, unlike TLS which secures a single application connection. It has two main modes: transport mode encrypts only the payload of each IP packet, typically used for host-to-host communication, while tunnel mode encrypts the entire original packet and wraps it in a new IP header, which is how site-to-site VPNs securely carry traffic across the public internet. Two protocols provide the actual protection: AH (Authentication Header) provides integrity and origin authentication but no encryption, while ESP (Encapsulating Security Payload) provides both encryption and authentication and is far more commonly used today. Before any protected traffic flows, IKE (Internet Key Exchange) negotiates a Security Association — the agreed algorithms, keys, and lifetimes — in a process conceptually similar to the TLS handshake but designed for peer-to-peer network-layer tunnels rather than a single client-server session.
- Secures all IP traffic between endpoints regardless of the application protocol
- Tunnel mode enables secure site-to-site VPNs over untrusted networks
- ESP provides both confidentiality and integrity for encapsulated packets
- IKE automates key negotiation and periodic rekeying without manual key exchange
AI Mentor Explanation
IPsec is like an armored transport service that carries an entire equipment crate — bats, pads, helmets, everything — sealed inside a locked container between two grounds, rather than just protecting one item at a time. Transport mode is like sealing only the valuables inside the existing kit bag, while tunnel mode is like putting the whole sealed kit bag inside a second, fully opaque crate for the journey. Before any crate moves, both grounds’ logistics teams agree on the lock type and shared key through a secure negotiation, exactly like IKE negotiating IPsec’s Security Association before ESP or AH protect the actual traffic.
Step-by-Step Explanation
Step 1
IKE negotiation
Peers negotiate algorithms, keys, and lifetimes to establish a Security Association before any protected traffic flows.
Step 2
Choose AH or ESP
AH provides integrity/authentication only; ESP (more common) provides encryption plus authentication.
Step 3
Choose mode
Transport mode encrypts just the payload; tunnel mode encapsulates the whole original packet under a new IP header.
Step 4
Protected transit
Packets traverse the network encrypted/authenticated per the negotiated Security Association until it expires and is rekeyed.
What Interviewer Expects
- Explains IPsec operates at the network layer, securing all traffic between endpoints
- Distinguishes AH (integrity only) from ESP (encryption plus integrity)
- Distinguishes transport mode from tunnel mode and when each is used
- Knows IKE negotiates the Security Association before data protection begins
Common Mistakes
- Confusing IPsec with TLS as if they operate at the same layer
- Thinking AH provides encryption, when it only provides authentication/integrity
- Not knowing tunnel mode is what makes site-to-site VPNs possible
- Forgetting that IKE, not the data protocols, handles key negotiation and rekeying
Best Answer (HR Friendly)
“IPsec is a set of protocols that encrypts and authenticates traffic at the network level, which is how most secure site-to-site VPNs work — imagine an armored truck route between two offices where every type of cargo automatically travels locked and verified, instead of securing just one application’s connection like a browser does with HTTPS. Before any traffic moves, both ends first agree on encryption keys, and then all subsequent data is protected under that arrangement until it is time to renegotiate.”
Code Example
# View active IPsec Security Associations (StrongSwan/Libreswan)
sudo ip xfrm state
# Example output shows the negotiated algorithm and mode:
# src 203.0.113.10 dst 203.0.113.20
# proto esp spi 0x00001234 reqid 1 mode tunnel
# auth-trunc hmac(sha256) ...
# enc cbc(aes)
# View configured IPsec tunnel policies
sudo ip xfrm policyFollow-up Questions
- What is the difference between IPsec transport mode and tunnel mode?
- How does IKE Phase 1 differ from IKE Phase 2 in negotiating a VPN tunnel?
- Why is ESP used far more often than AH in modern deployments?
- How does IPsec compare to a TLS-based VPN like OpenVPN or WireGuard?
MCQ Practice
1. At which layer does IPsec primarily operate?
IPsec secures IP packets at the network layer, protecting all traffic between two endpoints.
2. Which IPsec protocol provides both encryption and authentication?
ESP (Encapsulating Security Payload) provides confidentiality plus integrity; AH provides integrity only.
3. What does IPsec tunnel mode do to the original IP packet?
Tunnel mode wraps the whole original packet, including its header, inside a new outer IP packet.
Flash Cards
What is IPsec? — A protocol suite that authenticates and encrypts IP packets at the network layer, commonly used for VPNs.
AH vs ESP? — AH provides integrity/authentication only; ESP provides encryption plus authentication.
Transport vs tunnel mode? — Transport encrypts only the payload; tunnel encapsulates the entire original packet under a new IP header.
What negotiates IPsec keys? — IKE (Internet Key Exchange), which establishes the Security Association before data protection begins.