after a long night of beer blah hack splice graft perl miceql grunge splech pizza argh art blech … the database has been recovered.
There may be errors or lost data … please let the support or stuff or tech lists know.
Also the security updates were done on the raw machine … hold your breath!
Sonance technical services suffered a complete outage today until 13:30 or so when the machine was rebooted. Our thanks to Matthias for tramping through the snow to shiver for a while in the cold damp Keller poking at frozen iron.
Basic issue looks like an IRQ failure on the machine. Kernels were running (accepting connnections, returning pings) but no userland was getting any attention (no SSH, no keyboard on console!!!).
Note that the database failure of last weekend is still with us, so many user processes are broken (evolve, php applications). The basic issue here is that the passwords were all lost from the database, and we can’t figure out how to restore them from a backup as yet . . . We’ll look at it tonight at our meeting and try and script our way through the MySQL jungle.
Sonance suffered another housing-related outage again. According to Matthias G who went down to the Krypta, the machine has been moved from one place to another, and whoever moved the machine did not plug in the network cable.
Stuff that happens……..
Blogs up, evolve up, mailing lists up… Actually they never went down 