[HCoop-Discuss] [HCoop-Announce] Mire back up

[docelic]

On Tue, Feb 12, 2008 at 05:05:20PM +0100, Björn Lindström wrote:
> On Feb 12, 2008 4:40 PM, Michael Olson <mwolson at hcoop.net> wrote:
> > Our main webserver (mire) is back up and running.  The local root
> > exploit has been fixed.  We also took this opportunity to:
> >
> >  - Make Linux recognize all 5GB of RAM.
> >  - Tune some security options.
> >  - Upgrade the kernel to a newer 2.6.23.y maintenance release.
> 
> What happened with the suspected issue with the hard drive that was mentioned?

The hard drives and md arrays turned out to be fine.

The initial problem was that none of the kernels wanted to boot into
the system, not being able to mount the root device /dev/md0.

Upon examining the situation, it turned out that during the new kernel
installation, grub has removed the initrd lines from boot configuration.

As to why this happened, I have no idea.

One eventual possibility is, yaird (initrd image creator) did not want to
run at first; I had to upgrade it. It might have happened that, during that
first unsuccessful run, it messed up the configuration. 

Summing up all the experience from this outage, I will put together
a Wiki page documenting every step of the process, including notes
on manually inspecting all relevant files and configurations, no 
matter if they used to work in the past.

-doc