6 Ways to kill your Servers

Learning how to scale isn’t easy without any prior experience. Nowadays you have plenty of websites like highscalability.com to get some inspiration, but unfortunately there is no solution that fits all websites and needs. You still have to think on your own to find a concept that works for your requirements. So did I.
A few years ago, my bosses came to me and said “We’ve got a new project for you. It’s the relaunch of a website that has already 1 million users a month. You have to build the website and make sure we’ll be able to grow afterwards”. I was already an experienced coder, but not in these dimensions, so I had to start learning how to scale – the hard way.
The software behind the website was a PHP content management system, based on Smarty and MySQL. The first task was finding a proper hosting company who had the experience and would also manage the servers for us. After some research we found one, told them our requirements and ordered the suggested setup:

• LoadBalancer (+Fallback)
• 2 Webservers
• Mysql Server (+Fallback)
• development machine

They said, that’s gonna be all we need – and we believed it. What we got was:

• Loadbalancer (single core, 1GB RAM, Pound)
• 2 Webservers (Dual core, 4GB RAM, Apache)
• MySQL Server (Quad core, 8GB RAM)
• Dev (single core, 1GB RAM)

The setup was very basic without any further optimization. To synchronize the files (php+media files) they installed DRBD in active-active configuration.
Eventually the relaunch came – of course we were all excited. Very early in the morning we switched the domains to the new IPs, started our monitoring scripts and stared at the screens. We had almost instantly traffic on the machines and everything seemed to work pretty fine. The pages loaded quickly, MySQL was serving lots of queries and we were all happy.
Then, suddenly our telephones started to ring “We can’t access our website, what’s going on?”. We looked at our monitoring software and indeed – the servers were frozen and the site offline! Of course, the first thing we did was calling our hoster “hey, all our servers are dead. What’s going on?”. They promised to check the machines and call back immediately afterwards. The call came: “well,…erm… your filesystem is completely fubar. What did you do? It’s totally screwed”. They stopped the loadbalancers and told me have a look at one of the webservers. Looking at the index.php file I was shocked. It contained some weird fragments of C code, error messages and something that looked like log files. After some further investigation we found out that DRBD was the cause for this mess.

Lesson #1 learned

Put Smarty compile and template caches on an active-active DRBD cluster with high load and your servers will DIE!
While our hoster was fixing the webservers I rewrote some parts of the CMS to store the Smarty cache files on the servers local filesystems. Issue found & fixed. We went back online! Hurray!!!
Now it was early afternoon. The website usually reaches its peak in the late afternoon until early evening. At night the traffic goes back to almost none. We kept staring at the monitoring software and we were all sweating. The website was loading but the later it got, the higher the system load and the slower the responses. I increased the lifetime of the Smarty template caches and hoped it would do the trick – it didn’t! Very soon the servers started to give timeouts, white pages and error messages. The two machines couldn’t handle the load.
Our customer got a bit nervous at the same time, but he said: Ok, relaunches usually cause some issues. As long as you fix it quickly, it will be fine!
We needed a plan to reduce the load and discussed the issue with our hoster. On of their administrators came up with a good idea: “Guys, your servers are currently running on a pretty common Apache+mod_php setup. How about switching to an alternative webserver like Lighttpd? It’s a fairly small project, but even wikipedia is using it”. We agreed.

Lesson #2 learned

Put an out-of-the-box webserver configuration on your machines, do not optimize it at all and your servers will DIE!
The administrator gave his best and reconfigured both webservers as quickly as he could. He threw away the Apache configuration and switched to Lighttpd+FastCGI+Xcache. Later, when we went back online we almost couldn’t stand the pressure anymore. How long will the servers last this time?
The servers did surprisingly well. The load was MUCH lower than before and the average response time was good. After this huge relief we went home and got some sleep. It was already late and we came to the conclusion there was nothing left we could do.
The next days the website was doing rather well, but at peak times it was still close to crash. We spotted MySQL as the bottleneck and called our hoster again. They suggested a MySQL Master-Slave replication with a MySQL slave on each webserver.

Lesson #3 learned

Even a powerful database server has its limits and when you reach them – your servers will DIE!
In this case the database became so slow at some point, that the incoming and queued network connections killed our webservers – again. Unfortunately this issue wasn’t easy to fix. The content management system was pretty simple in this regard and there was no built-in support for separating reading and writing SQL queries. It took a while to rewrite everything, but the result was astonishing and worth every minute of suspended sleep.
The MySQL replication really did the trick and the website was finally stable! YEAH! Over the next weeks and months the website became a success and the number of users started to increase constantly. It was only a matter of time until the traffic would exceed our resources again.

Lesson #4 learned

Stop planning in advance and your servers are likely to DIE.
Fortunately we kept thinking and planning. We optimized the code, reduced the number of needed SQL queries per pageload and suddenly stumbled upon MemCached. At first I added MemCached support in some of the core functions, as well as in the most heavy (slow) functions. When we deployed the changes we couldn’t believe the results – it felt a bit like finding the Holy Grail. We reduced the number of queries per second by at least 50%. Instead of buying another webserver we decided to make even more use of MemCached.

Lesson #5 learned

Forget about caching and you will either waste a lot of money on hardware or your servers will die!
It turned out, that MemCached helped us to reduce the load on the MySQL servers by 70-80%, which also resulted in a huge performance boost – also on the webservers. The pages were loading much quicker!
Eventually our setup seemed to be perfect. Even on peak times we didn’t have to worry about crashes or slow responding pages anymore. Did we make it? No! Out of the blue one of the webservers started having some kinda hickups. Error messages, white pages and so on. The system load was fine and in most cases the server worked, but only in “most cases”.

Lesson #6 learned

Put a few hundred thousand small files in one folder, run out of Inodes and your server will die!
Yes you read it correct. We were so focused on MySQL, PHP and the webservers itself that we didn’t pay enough attention to the filesystem. The Smarty cache file were stored on the local filesystem – all in one single directory. The solution here was putting Smarty on a dedicated ReiserFS partition. Furthermore we enabled the Smarty “use_subdirs” option.
Over the past years we kept optimizing the pages. We put the Smarty caches into memcached, installed Varnish to reduce the I/O load for serving static files more quickly, switched to Nginx (Lighttpd randomly produced error 500 messages), installed more RAM, bought better hardware, more hardware… this list is endless.

Conclusion

Scaling a website is a never ending process. As soon as you fix one bottleneck you’r very likely to stumble into the next one. Never ever start thinking “that’s it, we’re done” and lean back. It will kill your servers and perhaps even your business. It’s a constant process of planning and learning. If you can’t get a job done on your own, because you have a lack of experience and/or resources – find a competent and realiable partner to work with. Never stop talking with your team and partners about the current issues and the ones that might arise in (near) future. Think ahead and be proactive!

Tips & Tricks

Listing running processes
You can list/view every process by memory and/or CPU on your system,using any of the following commands:
ps -ef
ps -e
ps -eF
ps -ely
List information for particular PIDs:
ps -p 1,2
List paths that the PID has opened:
lsof -p $

DNS cache cleaning
There is a simple command to quickly clean the DNS cache for every OS. On Linux, Make sure you have the nscd tool installed and running in the background as a daemon. In order to clear the DNS cache, simply restart the daemon as follows -- You need root privileges:
/etc/init.d/nscd restart

Live Interrupts Details
To watch the live interrupt changes in your system run:
watch -d 'cat /proc/interrupts'

Access a Windows Share from Bash
Ever wanted to access a Windows share from your terminal? Well, using mount and cifs/samba, this is possible. Make sure you have smbfs/cifs support. We need to make a directory on our hard disk where we can mount the Windows share.
mkdir /mnt/location
we are now ready to mount the filesystem on our newly created directory /mnt/location
To mount using cifs, use the following code:
mount -t cifs //server-ip-or-name/share /mnt/location -o username=user,password=pass,domain=DOMAIN
When we're done working on the share, We should exit the directory or close any programs that are accessing it, and then umount the windows share by using the following commands:
cd /
umount /mnt/location

Data Recovery in Ubuntu
Install ddrescue tools:
sudo apt-get install ddrescue
Connect the failed disk to your system:
Wait for a while...
We can now mount this image on our system and take a look at the files:
mount -t ext3 -o loop disk-image.img /mnt/tmp

How to install vmware/vmware Player on Ubuntu version*
sudo apt-get install build-essential linux-headers-$(uname -r)
sudo chmod +x VMware-Player*.bundle
sudo chmod +x VMware-Workstation-7.0.0-203739_i386-NoTools.bundle
sh VMware-Workstation-7.0.0-203739_i386-NoTools.bundle

Tunnel your SSH connection via intermediate host
$ ssh -t reachable_host ssh unreachable_host
This one-liner creates an ssh connection to unreachable_host via reachable_host. It does it by executing the ssh unreachable_host on reachable_host. The -t forces ssh to allocate a pseudo-tty, which is necessary for working interactively in the second ssh to unreachable_host.
This one-liner can be generalized. You can tunnel through arbitrary number of ssh servers:
$ ssh -t host1 ssh -t host2 ssh -t host3 ssh -t host4 ...

Clear the terminal screen
$ CTRL+l

Hear when the machine comes back online
$ ping -a IP
Ever had a situation when you need to know when the system comes up after a reboot? Up until now you probably launched ping and either followed the timeouts until the system came back, or left it running and occasionally checked its output to see if the host is up. But that is unnecessary, you can make ping -a audible! As soon as the host at IP is back, ping will beep!

Shutdown a Windows machine Remotely
$ net rpc shutdown -I IP_ADDRESS -U username%password

mtr - traceroute and ping combined
$ mtr google.com
MTR, bettern known as "Matt's Traceroute" combines both traceroute and ping command. After each successful hop, it sends a ping request to the found machine, this way it produces output of both traceroute and ping to better understand the quality of link. If it finds out a packet took an alternative route, it displays it, and by default it keeps updating the statistics so you knew what was going on in real time.

Copy your public-key to remote-machine for public-key authentication
$ ssh-copy-id remote-machine
This one-liner copies your public-key, that you generated with ssh-keygen (either SSHv1 file identity.pub or SSHv2 file id_rsa.pub) to the remote-machine and places it in ~/.ssh/authorized_keys file. This ensures that the next time you try to log into that machine, public-key authentication (commonly referred to as "passwordless authentication.") will be used instead of the regular password authentication.
If you wished to do it yourself, you'd have to take the following steps:
your-machine$ scp ~/.ssh/identity.pub remote-machine:
your-machine$ ssh remote-machine
remote-machine$ cat identity.pub >> ~/.ssh/authorized_keys
This one-liner saves a great deal of typing. Actually I just found out that there was a shorter way to do it:
your-machine$ ssh remote-machine 'cat >> .ssh/authorized_keys' < .ssh/identity.pub


Linux: Recovering deleted /etc/shadow password file

Sometimes by accident we may delete /etc/shadow  file. If you boot into single user mode, system will ask root password for maintenance, and just imagine you do not have a backup of /etc/shadow file. How do you fix such problem in a production environment where time is critical factor? Below is the explaination how to recover deleted /etc/shadow file in five easy steps. It will take around 10 min. to fix the problem.

Boot server into single user mode

1) Reboot server

2) Next, you will see grub-boot loader screen. Select Recovery mode the version of the kernel that you wish to boot and type e for edit. Select the line that starts with kernel and type e to edit the line.

3) Go to the end of the line and type init=/bin/bash as a separate one word (press the spacebar and then type init=/bin/bash). Press enter key to exit edit mode.
init=/bin/bash

4) Back at the GRUB screen, type b to boot into single user mode. This causes the system to boot the kernel and run /bin/bash instead of its standard init. This will allow us gain root privileges (w/o password) and a root shell.


 Make sure you can access system partition(s)

1) Mount partitions in read write mode
Since / is currently mounted read-only and many disk partitions have not been mounted yet, you must do the following to have a reasonably functioning system.
# mount -rw -o remount /
Do not forget to (re)mount your rest of all your partitions in read/write (rw) mode such as /usr /var etc (if any)


 Rebuild /etc/shadow file from /etc/passwd

1) You need to use pwconv command; it creates /etc/shadow from /etc/passwd and an optionally existing shadow.
# pwconv

2) Use passwd command to change root user password:
# passwd

Note you may need to type same password twice with passwd command. If you have an admin account, then setup password for that account. On most production, servers direct root login is disabled. In our situation, admin was the only account allowed to use su and sudo command.
# passwd admin

3) Now root and admin accounts are ready to go in multi-user mode. Reboot the system in full multiuser mode:
# sync
# reboot

Note:
    * Some time /etc/shadow- file can be use to replace /etc/shadow
    * If you have a backup of /etc/shadow on tape or cdrom then you can copy back /etc/shadow file

How to Step Block all non-root login

Block all non-root (normal) users until we fix all password related problems. Since rest of account do not have any password, it is necessary to prevent non-root users from logging into the system. You need to create /etc/nologin file, it will allow access only to root. Other users will be shown the contents of this file and their logins will denied (refused)

1) Login as root user (terminal login only)

2) Create /etc/nologin file
cat > /etc/nologin
System is down due to temporary problem. We will restore your access
within 30 minutes time. If you have any questions please contact tech
support at XXX-XXXX or techsupport@mycorp.com

Update all users password in batch mode

1) Create random password for each non-root user using chpasswd utility. It update passwords in batch mode. chpasswd reads a list of user name and password pairs from file and uses this information to update a group of existing users. Each line is of the format:

user_name:password

Remember by default the supplied password must be in clear-text format. This command is intended to be used in a large system environment where many accounts are created at a single time or in emergency like this. First, we need to find out all non-root accounts using awk command:
awk -F: '{ if ( $3 >1000 ) print $1}' /etc/passwd > /root/tmp.pass

Make sure /root/tmp.pass file contains non-root usernames only.

#############################################################################
How to figure out Differences between files and folders


Diff Between folders : diff --brief --recursive
Diff Between Files:   diff

## Iptables Internet Access to Private Network ###

On Mgmt Server
iptables -t nat -A POSTROUTING -o bond1 -j MASQUERADE
echo "1"> /proc/sys/net/ipv4/ip_forward

On Compute Node
route add default gw 10.10.20.254
cat /etc/resolv.conf
nameserver 10.10.20.254