[HCoop-Discuss] Alternative contact method

[Justin S. Leitgeb]

Jonathan Roes wrote:
>> If the servers were completely down, I'd pop into #hcoop @
>> irc.freenode.net to ask what was up, and probably get a login message
>> answering the question.
>>     
>
> I can imagine the situation where the servers are down and none of the
> admins are at their terminals and/or available on IRC for help.  I
> don't see this as too particularly unlikely.
>
> I think IRC is probably the best place for everyone to check on server
> status, with ops in the channel utilizing the topic or a ChanServ on
> join message to inform users of any downtime.  What might be a good
> idea is to have something automated that would verify that the server
> is responding to service requests, and when it is not responding
> within a certain threshold, notify administrators by their preferred
> emergency contact information - phone/cell, sms, etc.  This might even
> be a service offered by the colocation provider, I've seen it at other
> providers before.
>
>   
I don't know if Peer 1 would provide this for us, and I have a suspicion 
that they wouldn't.  However, I've seen other organizations use an 
off-site machine running nagios for monitoring purposes and I was 
planning on suggesting this to the hcoop-sysadmin list once the 
migration is under way.  If anyone has a machine that could run nagios 
for us off of the Peer 1 network, let us know! :)


> When an administrator begins taking a look, they could just drop a
> line in the topic to notify users of what's in progress.  Icing on top
> would be an IRC bot that automatically let us know an administrator
> has been notified by changing the topic.
>
>   
Nagios (and probably other open-source tools that I haven't researched) 
allows for triggers to external programs on specified events, so this 
should be pretty easy to implement.  We just need a machine on a stable 
connection that we can configure for this purpose.  I like the idea of 
connecting system monitoring with IRC. :)

- Justin