A Server Disk Has Failed: What to Do and What Not to Do

A disk failure in a server is a normal event, and if RAID is configured correctly, users usually do not even notice. The real danger lies elsewhere: an alert that goes unseen for weeks, a healthy disk pulled out instead of the failed one, or a hasty decision that destroys an array that could have been saved.

First: Understand the Situation Precisely

• Server management controller: iLO on HPE servers or iDRAC on Dell servers shows the status of disks and arrays.
• RAID controller tools: such as HPE Smart Storage Administrator, Dell OpenManage or storcli for Broadcom controllers.
• Software RAID: check an mdadm array with cat /proc/mdstat and ZFS with zpool status.
• SMART health: the command smartctl -a /dev/sda shows a disk’s errors and bad sectors.

The most important question is what state the array is in. Degraded means a disk has been lost but the data is still accessible. Failed or Offline means the array is no longer accessible, and you must proceed with far more caution.

Before Replacing: Confirm Your Backup

Rebuilding an array puts heavy load on all remaining disks, and that is exactly when the risk of a second disk failure rises. So check when the last successful backup was taken, and if it is old, take a fresh copy of critical data before replacing the disk. During the rebuild, postpone heavy workloads as much as possible.

Replacing the Disk Correctly

1. Get a compatible disk: the same type (SAS or SATA, HDD or SSD), with equal or greater capacity, and ideally the same model or one approved by the server manufacturer.
2. Identify the failed disk precisely: read the slot number from the management tool and use the Locate feature to make that disk’s light blink. Slot numbering is not always what it seems.
3. Hot-Swap or shutdown: if the server and controller support Hot-Swap, replace the disk while the server is running; otherwise, schedule a shutdown.
4. Start the rebuild: many hardware controllers start rebuilding automatically when they detect the new disk. With software RAID, you have to start it yourself; for example, with zpool replace in ZFS, or in mdadm, after partitioning the new disk, with mdadm --manage /dev/md0 --add /dev/sdb1.
5. Monitor rebuild progress until the array returns to a healthy state.

If the failed disk is a Proxmox boot disk on ZFS, in addition to replacing it in the pool you must also prepare the partitions and bootloader on the new disk according to the Proxmox documentation. Otherwise, the server may fail to boot when the next disk fails.

Never recreate the array (Create or Initialize) or clear the disk configuration in the RAID controller settings to “fix the problem”. If the controller reports a Foreign Configuration or the array has Failed, consult a specialist before confirming anything. One wrong choice can destroy data that could have been recovered.

If the Server Had No RAID

• Stop writing to the disk as much as possible.
• Do not run file system repair tools such as chkdsk on a failing disk; they can make things worse.
• If the disk can still be read, copy important data immediately or create an image of the whole disk with a tool such as ddrescue.
• If the disk makes unusual noises or is not detected, do not keep it powered on, and go to a data recovery specialist.

After the Problem Is Fixed

• Find out why the alert was not seen in time, and enable automatic alert delivery.
• If the disks in the array are the same age, keep a close eye on the rest of them.
• Get a replacement spare disk, and if you are returning the failed disk under warranty, think about the sensitive data on it.

Checklist

• Disk and array status alerts are sent automatically.
• A compatible spare disk is available.
• The latest backup was checked before the rebuild.
• The failed disk’s location was confirmed with Locate.
• The rebuild was monitored to completion and the array is healthy.

When a disk fails, ITPlus724 engineers can replace it and rebuild the array with minimal risk, and set up disk health monitoring for your servers.