There are several reasons.
If you forget to discover the new drive in the bios, your install is going right to the old drive.

.
If you have a Dell, they use "cable select" and a Cable Select IDE Cable (there is a small square hole punched in the ide cable). If you remember the old IBM computer punch cards, its a hole like that. The middle connector will be the master and the end one will be the slave. If you have only one drive and its hooked up wrong, no problem. Now hook up a second one and go for the format an/or install. You've probably just toasted your data.
It really gets wild if you decide to go from "cable select" to slave\master with your Dell "cable select" ide cable.
If you make a ghost of your drive with a floppy from DOS as a backup and you forget to remove the image drive before you boot the system back up, That drive will never be able to boot as a single, it wil error out. This is what I suspect happened to our original poster
Besides removing all other drives, you want to make sure, that printers, scanners, etc are also removed
Basically when you just address one drive, you eliminate the chance of tinkering with the wrong drive.