[Hcoop-discuss] Notes on server redundancy

[Justin S. Leitgeb]

These are some very important questions that we will definitely have to 
deal with as we grow.  Of course, with the next network architecture, 
initially only consisting of three servers, there will be numerous 
single points of failure.  Unfortunately, as Nathan pointed out, we 
probably can't do a whole lot about that before we get a bit bigger. 

We should really continue to talk about this, though, in order to scale 
out as gracefully as possible in the future.  Single points of failure 
may never be completely eliminated, but we can do a pretty good job of 
getting rid of them to increase resource availability.  Some data 
centers, just to give you some idea of what we can do, have up to 100 
load-balanced web servers in a single cluster, along with database 
clusters, redundant switches and fiber connections.  Furthermore, these 
things aren't all prohibitively expensive.  I put some preliminary 
comments on possibilities for Hcoop's future on the page 
http://wiki.hcoop.net/wiki/SystemArchitecturePlans (under "Scaling for 
Redundancy and Performance"), let's continue to talk about it there.

Justin

Nathan Kennedy wrote:

>Karl Chen wrote:
>  
>
>>The quote request letter looks fine to me.
>>
>>From the gripes page there's this bullet:
>>* If one machine goes down, then our services will go down, too.
>>
>>Is there anything not prohibitively expensive we can actually do
>>about this?  
>>
>>DNS can be redundant, mail can be queued, but anything that needs
>>files is screwed if the file server goes down...
>>  
>>    
>>
>Short of clustering, which is beyond our means at this point, I don't 
>think there is anything we can do about this.  I think what we want is 
>to prevent total hardware failure (with things such as redundant power 
>supplies, UPS--very nice if this is provided by the facility--and RAID), 
>and rapid on-site hardware support to replace failed hardware in the 
>event of outages.  If we do this right, the mean time between failures 
>may not be much worse than the lifetime of our hardware between upgrades 
>anyway.  In any event the great majority of commercial web hosts out 
>there function the same way--if a critical server kicks the bucket, the 
>sites it hosts go down until someone fixes it or restores from backup to 
>a replacement machine.  Truly clustered hosting is particularly 
>difficult to do with the kind of dynamic stuff that many of our users 
>run, and it is much more costly and complex to administer.   Once we get 
>a whole bunch of servers, say at least half a rack, then it could make 
>sense to keep a hot spare server idle on the rack, that could very 
>rapidly replace any failing servers.  That could reduce downtime and 
>also provide insurance since the probability of any one server failing 
>gets much higher as the number of servers increases.
>
>-ntk
>
>_______________________________________________
>Hcoop-discuss mailing list
>Hcoop-discuss at hcoop.net
>http://hcoop.net/cgi-bin/mailman/listinfo/hcoop-discuss
>  
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.hcoop.net/pipermail/hcoop-discuss/attachments/20060326/348c239f/attachment.htm