What is IPsec in 500 words?

IPsec is the most commonly used protocol to secure TCP/IP communications over 
an untrusted network. This basically works in two steps.

1) Setup an encrypted negotiation channel (called ISAKMP) to the other end and 
authenticate each other to prevent a man in the middle attack. 
2) Agree on the encryption parameters for the actual tunnel that will be used to send the encrypted data through. These parameters are for example which cipher
to use, what strength of the cipher to use (how many bits for the keys) or 
which algorithm to use. 

This negotiation is called the Internet Key Exchange protocol (IKE). When 
everything has been negotiated, the parties will attempt to set up the
promised connection, the "Security Association", or IPsec SA.

Once this negotiation has completed, encrypted packets can be send and 
received by both ends.

What is important to know when implementing this is the following:

1) Part of these negotiations happen over UDP port 500, or in some cases that
 involve NAT, over UDP port 4500. Be sure to allow these ports to communicate
 through your firewalls if you want to deploy IPsec.

2) The actual encrypted packets are not "normal IP" packets. People often think
   of "IP" as being TCP, UDP and ICMP, but there are many others. In fact, those
   are all IP packets with a different protocol number. ICMP is 1, TCP is 6 and
   UDP is 9. For IPsec, there are two new kinds of IP packets.

ESP: Encapsulated Security Payload. ESP has IP protocol number 50, and is used
     for almost all types of IPsec connections. This is also called "tunnel 
     mode".

AH:  Authentication Header. AH has protocol number 51, and is almost never used.
     The exception is Microsoft Windows, when it is using "L2TP". This is also
     called "transport mode".

You will also have to allow protocol 50 (and for L2TP protocol 51) packets
through your firewall. Do not confuse these numbers with port numbers! 
Allowing port 50 will not let ESP packets through. ESP packets have no port 
numbers at all!

If your external firewall is also your VPN server, you could use the following
rules, assuming ppp0 would be your interface to the internet:

iptables -I INPUT -p udp --dport 500,4500 -j ALLOW -i ppp0
iptables -I INPUT -p esp -j ALLOW -i ppp0
iptables -I OUTPUT -p udp --dport 500,4500 -j ALLOW -i ppp0
iptables -I OUTPUT -p esp -j ALLOW -i ppp0
# only needed for windows L2TP
iptables -I INPUT -p ah -j ALLOW -i ppp0
iptables -I OUTPUT -p ah -j ALLOW -i ppp0

If you get an error about iptables not knowing "esp" or "ah", you can write
it as a number instead. 50 for esp and 51 for ah.

If your firewall and VPN server are not the same, be careful to use the
FORWARD tables instead of the INPUT/OUTPUT tables. Also be aware of NAT on
your firewalls. You CANNOT use NAT on IPsec packets. For more about NAT,
see LINK

Most of these negotiations all happen "under the hood". Don't worry if this all
seems complex. Configuration is actually fairly easy.