Spent the last week and this week on a personal Ruby on Rails project. This involves subversion (as a version management system), mongrel, capistrano, ftp, postgresql, some smarts with DNS, exim and a two-day complete re-install of Debian. That re-install was not expected.
Unix has this wonderful and powerful concept: the root user knows what they are doing at all times. 99.9% of the time this is a safe assumption. 0.1% of the time you type “yes” instead of “no” – removing the kernel in this fashion is highly not recommended.
How do you fix a broken Linux install?
Stage one involved making what is known as a LiveCD, or bootable Linux. I decided to download and boot from a Knoppix LiveCD. A quick restart from the CD, and I could see that the data was intact.
Stage two was installing a new 350GB HD for the data to bring the server up to 0.5TB of storage. The old faithful Unix standbys of dd, fsck from the old 200GB to the new 350GB and start the difficult work.
Stage three is a full Debian reinstall onto the old 200GB with 0.5Gb download of the most current packages, and re-apt on a 686 rather than 386 kernel. This didn’t take too long. Re-configuring all the servers and services: dns, dhcp, CUPS, Samba, Apache, subversion, rails+gems, python took most of the weekend.
Stage four: backup scripts. 0.5TB is too much of a mountain of data to lose.