[HCoop-Discuss] [HCoop-Announce] Mire back up

[Nathan Kennedy]

First I would echo what Davor said in the responsse to Steve.  Although I
think he was being rather modest and restrained.

> On 2/12/08, Björn Lindström <bkhl at hcoop.net> wrote:
>> Your idea that the fact that you are paying in money to the coop would
>> imply some additional obligations for the admins is invalid. They are
>> after all members of the coop just like you.
>
> I am not assuming this.  But those who volunteer to be an admin, need
> to treat it like a job, just because it is a volunteer position, does
> not mean it is not important.

Our admins have all, for the most part, treated their responsibilities as
volunteers as very important and have devoted a lot of energy to resolving
downtime in a timely manner.

You cannot expect them to treat it just like a full-time job, because our
admins have full-time jobs.  We are not paying them, and as you yourself
mention you have (or claim to have) the skills to volunteer as an admin
but would rather not contribute.

We all agree that this downtime was unacceptable.  Davor spent more than
half of a day working pretty much continuously on it.  Much of the delay
(other than some hours last night where people were sleeping, but which
was precipitated in any event by this delay) was due to a password change
for security purposes after one of our main admins resigned, which had not
been fully propagated to all of our current, understaffed admins.  In fact
Adam Megacz actually did jump in to help out a bit even though he is no
longer serving as an admin.  But in any event it was not only a simple
problem of grub configuration.  It was exacerbated by unfortunate timing
of the KVM password change for security reasons and by keyboard BIOS
problems with Mire.

That doesn't make this unacceptable, but it has to be put in perspective. 
When we were just on fyodor or abulafia we had situations like this where
we had to put a new kernel or hardware in and we ended up with downtime
for days on end, and ALL services were down.  We have improved this. 
Deleuze was up, email and DNS were served with no interruption, the portal
and all mail services were accessible throughout.  We have solid plans to
provide more redundancy in the future.

So while complaints are justified, individual blame without suggestions
for improvement or offers of assistance (colloquially, "bitching and
moaning") is not terribly helpful here.

Also worth noting is what Davor pointed out about our openness.  As is
well known, purely commercial providers have problems too.  Just yesterday
there was a system-wide, multiple hour total outage of RIM's Blackberry
service.  They must have hundreds of full-time, well-paid engineers, but
they had a total outage to millions of customers for hours.  All we know
is that they had unspecified "hardware problems" that they promise will
"not be repeated" (although that was heard twice before in the past year).

So I wish that I could promise that this incident will "never be
repeated," but instead I will say that this specific incident has been
solved, that our volunteers put a lot of work into getting it solved, for
free for our members benefit, that we have definite plans in the pipeline
for making such issues less likely in the future and easier to resolve,
and that we are still looking for additional admins to take on some
responsibilities.

When we do grow to the point where we have hired staff, it will be
appropriate to "blame" such persons for lack of zeal in the performance of
duty.  I don't think that is appropriate for our admins who not only put
in the amount of work they do for no compensation, with a good deal of
competence and diligence, but also in the process set themselves up as
targets for these complaints when they donate their time.  I don't think
you can point to a place where we claim to current or prospective members
that we have paid staff available at all times, because we don't.  If we
did, we would be billing at a much higher rate for our size now.

-ntk