Your server is running very slow.
Top looks Okay. Loads are not insane. Ram is high, but not out of bounds. sar shows high i/o wait times. Swap usage is not an issue. MySql process show hundreds of sleeping processes. Restarting MySql or Apache clears them, but then they start right back up.
WHAT? This make no sense!
tail /var/www/vhost/domain.com/statistics/log/access_log
…
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/mlomeupenvtb2012tb201212tb201212044071d032736e44d9b3e5b914d378f9e2jpg.jpg HTTP/1.0″ 200 16322 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/pics2dsstaticcomprodimg165178300jpg.jpg HTTP/1.0″ 200 12690 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/slimagesmacyscomisimageMCYproducts4optimized515264fpxtif.jpg HTTP/1.0″ 200 10497 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/plefuxcom6120111219A0361000WNipadiphonebatteriesexternal5000mah3751965bigjpg.jpg HTTP/1.0″ 200 9638 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/taylorgiftscomimagesp43126500jpg.jpg HTTP/1.0″ 200 59977 “-” “-”
Notice how these connections are coming from the server itself instead of from an external IP.
Now look at who is connecting to the server:
netstat -nat | grep :80 | gawk '{ print $5; }' | gawk -F: '{ print $1 }' | sort | uniq -c | sort -n
2 66.249.73.222
3 157.55.32.143
3 199.30.20.68
3 199.30.20.76
4 131.253.24.85
4 199.30.20.106
4 23.67.252.11
4 65.55.55.229
5 174.125.28.4
12 23.67.252.59
325 64.150.184.165
Again, all coming from the server. The solution to the problem was discovered in /tmp
ls -la /tmp
total 44532
drwxrwxrwx 4 root root 3522560 Dec 7 17:12 .
drwxr-xr-x 24 root root 4096 Dec 6 13:03 ..
drwx–x–x 2 apache apache 4096 Feb 29 2012 .bash
-rw-r–r– 1 apache apache 37281 Oct 13 10:21 .dsf
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.1
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.2
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.3
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.4
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.5
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.6
-rw-r–r– 1 apache apache 37281 Oct 13 18:18 .dsf.7
-rw-r–r– 1 apache apache 37281 Oct 13 18:18 .dsf.8
now,
ls -la /tmp/.bash
total 27392
drwx–x–x 2 apache apache 4096 Feb 29 2012 .
drwxrwxrwx 4 root root 3522560 Dec 7 17:14 ..
-rwx–x–x 1 apache apache 146 Nov 12 2012 1
-rwxr-xr-x 1 apache apache 323 Jan 13 2011 autorun
-rwx–x–x 1 apache apache 8922 Jan 23 2006 b
-rwx–x–x 1 apache apache 19557 May 9 2005 b2
-rwxr-xr-x 1 apache apache 11445 Jan 5 2011 bang
-rwxr-xr-x 1 apache apache 12321980 Feb 29 2012 bangnew
-rwxr-xr-x 1 apache apache 11824732 Jan 23 2011 bangold
-rw-r–r– 1 apache apache 44 Aug 3 03:28 cron.d
-rwx–x–x 1 apache apache 14679 Nov 2 2005 f4
-rwxr-xr-x 1 apache apache 15988 Sep 7 2002 juno
-rw-r–r– 1 apache apache 11 Aug 3 03:28 mech.dir
-rwx–x–x 1 apache apache 566 Jan 20 2013 mech.set
-rwxr-xr-x 1 apache apache 27 Jan 11 2011 run
-rwx–x–x 1 apache apache 152108 Jan 11 2011 sshd:
-rwxr-xr-x 1 apache apache 17 Nov 5 2008 start
-rwxr-xr-x 1 apache apache 8231 Feb 29 2012 std
-rwxr-xr-x 1 apache apache 13399 Aug 6 2000 stealth
-rwx–x–x 1 apache apache 8790 Jan 23 2006 stream
-rwxr-xr-x 1 apache apache 17690 Feb 6 1996 synk
-rwxr-xr-x 1 apache apache 6442 Jun 23 2011 talk
-rwxr–r– 1 apache apache 166 Aug 3 03:28 update
-rwx–x–x 1 apache apache 14841 Jul 22 2005 v
-rwxr-xr-x 1 apache apache 14911 Mar 6 2002 v2
End Result
End result: This server ahs been root compromised. The only solution is to reinstall and slave drive the existing compromised drive.