[Hcoop-discuss] Next hardware configuration/Email service

[Justin S. Leitgeb]

Adam Chlipala wrote:

>Maybe we have different ideas of what a "proper" back-up is.  I think 
>it's OK for any IMAP files that are being modified to be lost or 
>corrupted in the backup images, assuming each message gets its own 
>file.  (Most messages are only modified at receipt time, so we'll catch 
>them in the next backup.)  We are currently backing up all mailboxes 
>on-disk with rsnapshot (a script based on rsync), and I'm not aware of 
>any IMAP-specific correctness problems with the scheme.  Are you just 
>worried about performance?
>  
>

No, I guess I was just worried about lost messages and (possibly) 
corrupted database files.  I don't have a ton of experience on loaded 
IMAP systems, other than the personal server I run at home for 
convenience, and the reading I've done on the web.  There is a good page 
on backing up cyrus IMAP servers on the Cyrus wiki:

http://acs-wiki.andrew.cmu.edu/twiki/bin/view/Cyrus/Backup

Maybe as our volume increases, and we want to do things the "correct" 
way these methods may be helpful... right now I know I don't care enough 
about a lost message or two to fix it on fyodor myself.

>Like I said before, things are easier if we only have to apply "heavy 
>duty" backup techniques to a single filesystem.  This is why it's 
>preferable to back up IMAP files to the shared filesystem, if that's not 
>their primary home.
>
>  
>
Yeah, this should work -- we could stop the server for 5 minutes while 
we get an rsync snapshot, then bring it back up, then back up the 
finished copy made with rsync.

>Justin S. Leitgeb wrote:
>
>  
>
>>The point is that AFS alone is not going to deal with performance 
>>issues.  Real-world web sites can stress even large machines by today's 
>>standards.  I just think that we need to recognize this out front, and 
>>realize that we can't scrimp on the disk configuration of the fileserver 
>>-- RAID 10 may be something we should consider if we really want to go 
>>down that route, and it won't be cheap.  We will also want loads of 
>>memory in the front-end servers so that they can cache rather than 
>>introducing additional overhead assembling AFS packets.
>>
>>    
>>
>We could do this in a forward-looking way by putting in the 
>infrastructure for this sort of organization, but just not including 
>much capacity.  A simple proof that this would work to start out with is 
>the fact that we get by fine with our current server specs, and in fact 
>we are underutilizing what we have.  Maybe RAID 10 and RAM are not 
>cheap, but we wouldn't need much of them to start out with.  We'd still 
>be able to gain valuable experience using them in our initially 
>undemanding setting.
>
>In any case, it's probably a good idea to pick out a few most promising 
>hardware configurations, price them out, and see what seems worth the cost.
>  
>

Sounds good.

Justin