Thursday, April 07, 2005

Making Sense of tcp_tw_reuse

So, you've killed a server process (e.g. tomcat) on your Linux host and it won't restart immediately because it can't bind port 8180 (or whatever). Here's what's happening, I think:
  • TCP requires that the side of a "connection" that requested a teardown (first to send a FIN) hang around for 2*MSL (4 minutes, as MSL is 2 minutes) in TIME-WAIT.
  • Out of the box, most UNIXen will not allow a process to bind a TCP port that has another connection still active on it, regardless of where the process intends to connect; presumably this is about enforcing the TCP TIME-WAIT state and/or about constraining (to 1) the number of processes that can listen on a port.
  • So, if you kill a process which tends to behave as a client (e.g. Mozilla) while it has a connection open, the connection will hang around in TIME-WAIT for 4 minutes. This will not upset a restarted Mozilla because when it binds ports for making connections as a client it allows the OS to select a(nother) port for it.
  • On the other hand, if you kill a server process while it has open connections to the port(s) that it listens on, those ports themselves (e.g. 8180 for tomcat) are party to connections that the server host is responsible for shutting down (the server process was killed, rather than, say, the client sent a FIN) and will consequently have TIME-WAIT connections hanging around for 4 minutes. As the server process will want to bind the same port(s) again upon restart, there will be a problem.
So, to get around this,
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
The kernel documentation says:

Allow to reuse TIME-WAIT sockets for new connections when it is
safe from protocol viewpoint. Default value is 0.
It should not be changed without advice/request of technical
experts.
There is also tcp_tw_recycle:
Enable fast recycling TIME-WAIT sockets. Default value is 0.
It should not be changed without advice/request of technical
experts.
which looks to me like an "ignore 2MSL altogether" and goes further than is neccessary for the solving of this problem.