[Hcoop-discuss] rsnapshot errors

[Ryan M.]

There must be a that is too large, even for swap.  while cp doesn't put everything into memory that it copies, linux does store a good deal of what it accesses into cache.  Depending on how the Linux vm proceedes onces the low memory limit (on 2.6. kernels ONLY, /proc/sys/vm/min_free_kbytes) is reached, that could spill over into swap or the cache could be relieved as cp continues its job and therefore physical memory not ever running out. There IS a min_free_kbytes somewhere in the 2.4 kernel -- there has to be, but I have no idea _how_ the vm uses it to make decisions about what to do with overages in memory use.  My vm in 2.6 very well behaved, that's all I can say.  Copies never touch swap, but do drain physical memory by over 300mb when the operation is on an unclean kernel tree (345mb large).  

Take a look at the output of /proc/meminfo (maybe even save it) over a period of time during the backup process.  Or look at top, but that isn't a very thorough representation of what is going on.  Note that we have, at the moment, only 34mb of physical memory available.  In order to do what the kernel desires, it should spill over into swap.  But taking 3gb in a single copy and the vm eating every bit of swap... that can't be? :-)  Or is it complaining that it cannot read x into memory because it is larger than the 34mb available?  How does the 2.4 vm work?

I'm guessing that recently there has been more disk use wrt space taken?  Can someone test my theory?  Can we split the cp operations?  Once the cp is finished, and a new process takes over, the cached / swapped data should be purged.  IF this is the case, we can have it count how much it has already read, and stop at a certain amount (say, 300mb), then start over again from where it left off.  


Ryan


On Fri, 23 Dec 2005 08:23:40 -0800, AC wrote about "[Hcoop-discuss] rsnapshot errors" :

> Lately, the rsnapshot backup program has been crashing like this, so we 
> don't actually have any especially recent backups.  Does anyone have any 
> idea how we could get around the error, which I've included below?
> 
> /bin/cp: memory exhausted
> ---------------------------------------------------------------------------- 
> 
> rsnapshot encountered an error! The program was invoked with these options:
> /usr/bin/rsnapshot daily 
> ---------------------------------------------------------------------------- 
> 
> ERROR: /bin/cp failed. Perhaps this is not GNU cp?
> ERROR: Error! cp_al("/home/backup/rsnapshot/daily.0/", 
> "/home/backup/rsnapshot/daily.1/")
> 
> _______________________________________________
> Hcoop-discuss mailing list
> Hcoop-discuss at hcoop.net
> http://hcoop.net/cgi-bin/mailman/listinfo/hcoop-discuss