Creating a Hot Spare Server for the Cobalt Raq550 Brought to you by Internet Workflow Systems, LLC

WARNING: Be sure to apply all updates prior to beginning this procedure. In particular, both machines must have all patches so that they match. UNMATCHED SYSTEMS MAY NOT HAVE STABLE RESULTS!

Automated Setup

This process allows you to check out a set of scripts from our CVS repository and then run the setup program. This is the easiest way to accomplish a Hot Spare Server; however, there are a few reasons why you would NOT want to do this.

• You are not running the default configuration of the Raq550 OS
• You would rather create a derivative of this procedure
• You would like to know exactly what is being done to your server
• You would like the practice of setting this up manually
• If litigation is a hobby of yours and you would consider legal action against us if it doesn't work exactly the way you expect. (Consider this a disclaimer)

Primary System

1. Telnet or SSH into the Raq550 server.
2. Super Use to the root user
   su - root
3. Verify you are in the /root directory
   pwd
4. Login to the CVS repository
   cvs -d :pserver:anonymous@cvs.internetworkflow.com:/usr/cvsroot login
   password: anoncvs
5. Checkout the setup scripts from the repository
   cvs -d :pserver:anonymous@cvs.internetworkflow.com:/usr/cvsroot checkout Raq550/HotSpare
6. Execute the setup script for the primary server
   ./Raq550/HotSpare/setup primary

Failover System

1. Repeat the Steps 1-5 of the Automated Setup for the Failover Server above
2. Execute the setup script for the failover server
   ./Raq550/HotSpare/setup failover

Manual Setup

This process walks you through the manual creation of a Raq550 Hot Spare system. This procedure assumes some knowledge of how to get around in linux as copying, moving and editing files is required. If you don't know how to do this stuff, you had better just stick with the automated version above.

Primary System

Basically, this installs the rsync service so that the failover machine can synchronize with it. Other than that, the primary machine is left untouched so as to be as unintrusive as possible.

1. Add the following line to /etc/inetd.conf
rsync stream tcp nowait root /usr/bin/rsync rsync -daemon
2. Create the file /etc/rsyncd.conf
   The IP Address in this example should be replaced with the IP for your Failover Server
   Example:
   

   #/etc/rsyncd.conf
   max connections = 5
   log file = /var/log/rsync.log
   timeout = 300
   strict modes = false
   
   [home]
   	comment = Home Directory
   	path = /home
   	read only = yes
   	list = yes 
   	uid = root
   	gid = root
   	hosts allow = {IPADDR}
   	hosts deny = *
   
   [etc]
   	comment = Configuration Directory
   	path = /etc
   	read only = yes
   	list = yes
   	uid = root
   	gid = root
   	hosts allow = {IPADDR}
   	hosts deny = *
   
   [var]
           comment = Var Directory
           path = /var
           read only = yes
           list = yes
           uid = root
           gid = root
           hosts allow = {IPADDR}
           hosts deny = *
   
   [usr]
           comment = User Directory
           path = /usr
           read only = yes
           list = yes
           uid = root
           gid = root
           hosts allow = {IPADDR}
           hosts deny = *
   
   

3. Restart the inet service
   /etc/rc.d/init.d/inet restart
4. Set the max number of filehandles to larger than the default by adding the following lines to /etc/rc.d/rc.sysctl just before the exit 0 command:
   

   # we want to set the max number of file handles to 32K
   if [ -e /proc/sys/fs/file-max ]; then
   	echo 32768 > /proc/sys/fs/file-max
   fi
   exit 0	
   

Failover System

1. Create the file /bin/synchronize.sh
   The IP Address in this example should be replaced with the IP for your Primary Server
   Example:
   

   #!/bin/bash
   echo "Starting Synchronization `date`"
   echo "Synchronizing Home Directory"
   rsync -aHlpogDtvz --delete rsync://root@{IPADDR}/home /home
   echo "Synchronizing usr Directory"
   rsync -aHlpogDtvz --delete rsync://root@{IPADDR}/usr /usr
   echo "Synchronizing etc Directory"
   rsync -aHlpogDtvz --delete rsync://root@{IPADDR}/etc /etc.backup
   echo "Synchronizing var Directory"
   rsync -aHlpogDtvz --delete --exclude=log rsync://root@{IPADDR}/var /var
   echo "Synchronization Complete `date`"
   
   

2. Move the /etc directory to be /etc.orig
mv -f /etc /etc.orig
3. Create a Symbolic Link for the /etc directory
   ln -s /etc.orig /etc
4. Create the /etc.backup directory
   mkdir /etc.backup
5. Create the file /bin/failover
   Example:

   #!/bin/bash
   rm /etc
   ln -s /etc.backup /etc
   reboot
   

6. Create the file /etc/logrotate.d/rsync
   Example:

   /var/log/rsync.log {
     notifempty
     weekly
     rotate 3
     missingok
     compress
   }
   

7. Create the file /etc/cron.hourly/synchronize
   Example:

   nohup synchronize.sh >>/var/log/rsync.log 2>&1 &
   

8. Set Permissions for Executables
   chmod 755 /bin/failover
chmod 755 /bin/synchronize.sh
chmod 755 /etc/cron.hourly/synchronize

Initiating and Recovering from a Failover Condition

see Cobalt Raq550 Failover & Recovery