# $Id: tcp_tunning.txt,v 1.1 2007/01/17 22:14:43 andrei Exp $
#
# History:
# --------
# 2006-01-26  created by andrei

TCP Tunning/monitoring for lots of open connections

0. Introduction
----------------

This document describes very briefly various settings that should improve
 ser+tcp performance for sites handling a lot of tcp traffic (> 1000 open
  connections or very high connection/disconnection rates).
For now it deals only with linux specific optimizations.


1. Usefull kernel settings
---------------------------

1.1 connection rate/pending connections: by default the connection rate is
 too small

net.core.somaxconn             -  limit of the listen() backlog, default 128
net.ipv4.tcp_max_syn_backlog   -  default 1024 or 128
net.ipv4.tcp_timestamps        - default on., should be on (along with 
                                 tcp_tw_recycle and timestamp supporting
                                 peers allows for fast connections rates)

1.2 connection in close_wait: connection should stay as little as possible
 in close_wait to quickly free the fd/resources for new connections attempts
 WARNING: this could break normal tcp use, use it only if you know what you are
  doing
 
net.ipv4.tcp_max_tw_buckets - maximum number of timewait sockets
                              (the default seems to be ok)
net.ipv4.tcp_tw_recycle     - enables fast time wait sockets recycling (default
                              off), should be enabled if you have lots of short
                              lived connections 
                              WARNING: see the above warning
net.ipv4.tcp_tw_reuse       - allows reusing of time-wait sockets (default off)
                              WARNING: see above

net.ipv4.tcp_syncookies     - default off, in this case it's probably better to
                              keep it off

1.3 port range
net.ipv4.ip_local_port_range - should be increased (e.g. 4096-65534)

1.4  open fds
fs.file-max                 - maximum number of fds that will be allocated
                              (you probably need to increase it, default 
                               depends on installed memory)

1.5 other sysctl that might affect tcp connection rate or the maximum number
    of open connections

iptables  - remove the ip_conntrack module (it limits the maximum tcp
 connections, adds extra overhead (slow)). It's probably better to remove
  all the iptables modules.

net.ipv4.tcp_max_orphans    - might be worth a look if things go wrong
net.core.netdev_max_backlog - maximum device backlog

ifconfig <dev> txqueuelen <val> - set device transmission queue len


2. Monitoring (values to watch for)
-----------------------------------

2.1 FDs

fs.dentry-state  - format: nr. dentries, nr. unused, age_limit, want_pages
fs.file-nr       - format: allocated, unused, max (==fs.file-max)
fs.inode-state  -  format: nr. allocated, nr. free, preshrink

2.2 TCP

/proc/net/netstat - the TW, TWRecycled, TWKilled, PAWPassive, PAWActive,
                    PASWEstab fields
                   ( cat /proc/net/netstat |cut -d" " -f12-17 ;
                     cat /proc/net/sockstat)
/proc/net/sockstat


3. Ser settings

- don't forget to increase tcp_max_connections and the amount of shared memory
- you should increase the number of ser "tcp_children" processes (-N no)
As a rule of thumb, (maximum simultaneous connections)/2000 should be ok
- you might have to decrease TCP_BUF_SIZE to a smaller value (e.g 8K)
- you might want to increase PKG_MEM_POOL_SIZE (for large queues)

- you might need to increase the maximum open fds limit before starting ser
 (e.g. ulimit -n 1000000)

