Bootstrapping Vinum: A Foundation for Reliable Servers
Prev |  | Next

# 4. Failure Scenarios

This section contains descriptions of various failure scenarios. For each scenario, there is a subsection on how to configure your server for degraded mode operation, how to recover from the failure, how to exit degraded mode, and how to simulate the failure.

> Tip: Make a hard copy of these instructions and leave them inside the CPU case, being careful not to interfere with ventilation.

## 4.1. Root filesystem on ad0 unusable, rest of drive ok

> Note: We assume here that the boot blocks and disk label on `/dev/ad0` are ok. If your BIOS can boot from a drive other than `C:`, you may be able to get around this limitation.

### 4.1.1. Configure Server for Degraded Mode

1. Use BootMgr to load kernel from `/dev/ad2s1a`.

   1. Hit F5 in BootMgr to select `Drive 1`.

   2. Hit F1 to select `FreeBSD`.

2. After the kernel is loaded, hit any key but enter to interrupt the boot sequence. Boot into single-user mode and allow explicit entry of a root filesystem.

       Hit [Enter] to boot immediately, or any other key for command prompt.
       Booting [kernel] in 8 seconds...

       Type '?' for a list of commands, 'help' for more detailed help.
       ok boot -as

3. Select `/rootback` as your root filesystem.

       Manual root filesystem specification:
           <fstype>:<device>  Mount <device> using filesystem <fstype>
       			 e.g. ufs:/dev/da0s1a
           ?                  List valid disk boot devices
           <empty line>       Abort manual input

         mountroot> ufs:/dev/ad2s1a

4. Now that you are in single-user mode, change `/etc/fstab` to avoid the bad root filesystem.

   > Tip: If you used the `bootvinum` Perl script from Appendix A below, then these commands should configure your server for degraded mode.

   >     # fsck -p /
   >     # mount /
   >     # cd /etc
   >     # mv fstab fstab.bak
   >     # cp fstab_ad0s1_root_bad fstab
   >     # cd /
   >     # mount -o ro /
   >     # vinum start
   >     # fsck -p
   >     # ^D

### 4.1.2. Recovery

1. Restore `/dev/ad0s1a` from backups or copy `/rootback` to it with these commands:

       # umount /rootbad
       # newfs /dev/ad0s1a
       # tunefs -n enable /dev/ad0s1a
       # mount /rootbad
       # cd /rootbad
       # dump 0f - / | restore rf -
       # rm restoresymtable

### 4.1.3. Exiting Degraded Mode

1. Enter single-user mode.

       # shutdown now

2. Put `/etc/fstab` back to normal and reboot.

       # cd /rootbad/etc
       # rm fstab
       # mv fstab.bak fstab
       # reboot

3. Reboot and hit F1 to boot from `/dev/ad0` when prompted by BootMgr.

### 4.1.4. Simulation

This kind of failure can be simulated by shutting down to single-user mode and then booting as shown above in Section 4.1.1.

## 4.2. Drive ad2 Fails

This section deals with the total failure of `/dev/ad2`.

### 4.2.1. Configure Server for Degraded Mode

1. After the kernel is loaded, hit any key but Enter to interrupt the boot sequence. Boot into single-user mode.

       Hit [Enter] to boot immediately, or any other key for command prompt.
       Booting [kernel] in 8 seconds...

       Type '?' for a list of commands, 'help' for more detailed help.
       ok boot -s

2. Change `/etc/fstab` to avoid the bad drive. If you used the `bootvinum` Perl script from Appendix A below, then these commands should configure your server for degraded mode.

       # fsck -p /
       # mount /
       # cd /etc
       # mv fstab fstab.bak
       # cp fstab_only_have_ad0s1 fstab
       # cd /
       # mount -o ro /
       # vinum start
       # fsck -p
       # ^D

   If you do not have modified versions of `/etc/fstab` that are ready for use, then you can use `ed` to make one. Alternatively, you can `fsck` and `mount` `/usr` and then use your favorite editor.

### 4.2.2. Recovery

We assume here that your server is up and running multi-user in degraded mode on just `/dev/ad0` and that you have a new spindle now on `/dev/ad2` ready to go.

You will need a new spindle with enough room to hold root and swap partitions plus a Vinum partition large enough to hold `/home` and `/usr`.

1. Create a BIOS partition (slice) on the new spindle.

       # /stand/sysinstall

   1. Select `Custom`.

   2. Select `Partition`.

   3. Select `ad2`.

   4. Create a FreeBSD (type 165) slice large enough to hold everything mentioned above.

   5. Write changes.

   6. Yes, you are absolutely sure.

   7. Select BootMgr.

   8. Quit Partitioning.

   9. Exit `/stand/sysinstall`.

2. Create disk label partitioning based on current `/dev/ad0` partitioning.

       # disklabel ad0 > /tmp/ad0
       # disklabel -e ad2

   This will drop you into your favorite editor.

   1. Copy the lines for the `a` and `b` partitions from `/tmp/ad0` to the `ad2` disklabel.

   2. Add the `size` of the `a` and `b` partitions to find the proper `offset` for the `h` partition.

   3. Subtract this `offset` from the `size` of the `c` partition to find the proper `size` for the `h` partition.

   4. Define an `h` partition with the `size` and `offset` calculated above.

   5. Set the `fstype` column to `vinum`.

   6. Save the file and quit your editor.

3. Tell Vinum about the new drive.

   1. Ask Vinum to start an editor with a copy of the current configuration.

          # vinum create

   2. Uncomment the drive line referring to drive `UpWindow` and set `device` to `/dev/ad2s1h`.

   3. Save the file and quit your editor.

4. Now that Vinum has two spindles again, revive the mirrors.

       # vinum start -w usr.p1.s0
       # vinum start -w home.p1.s0

5. Now we need to restore `/rootback` to a current copy of the root filesystem. These commands will accomplish this.

       # newfs /dev/ad2s1a
       # tunefs -n enable /dev/ad2s1a
       # mount /dev/ad2s1a /mnt
       # cd /mnt
       # dump 0f - / | restore rf -
       # rm restoresymtable
       # cd /
       # umount /mnt

### 4.2.3. Exiting Degraded Mode

1. Enter single-user mode.

       # shutdown now

2. Return `/etc/fstab` to its normal state and reboot.

       # cd /etc
       # rm fstab
       # mv fstab.bak fstab
       # reboot

### 4.2.4. Simulation

You can simulate this kind of failure by unplugging `/dev/ad2`, write-protecting it, or by this procedure:

1. Shutdown to single-user mode.

2. Unmount all non-root filesystems.

3. Clobber any existing Vinum configuration and partitioning on `/dev/ad2`.

       # vinum stop
       # dd if=/dev/zero of=/dev/ad2s1h count=512
       # dd if=/dev/zero of=/dev/ad2 count=512

## 4.3. Drive ad0 Fails

Some BIOSes can boot from drive 1 or drive 2 (often called `C:` or `D:`), while others can boot only from drive 1. If your BIOS can boot from either, the fastest road to recovery might be to boot directly from `/dev/ad2` in single-user mode and install `/etc/fstab_only_have_ad2s1` as `/etc/fstab`. You would then have to adapt the `/dev/ad2` failure recovery instructions from Section 4.2.2 above.

If your BIOS can only boot from drive one, then you will have to unplug drive `YouCrazy` from the controller for `/dev/ad2` and plug it into the controller for `/dev/ad0`. Then continue with the instructions for `/dev/ad2` failure recovery in Section 4.2.2 above.

Prev | Home | Next
Where to Go from Here? |  | bootvinum Perl Script