[Hcoop-discuss] rsnapshot errors

[Karl Chen]

When I look at 'top' during the recurring backups, I usually see
cp using 100s of MB of RAM and I'm not surprised it exceeded some
limit somewhere.  Perhaps it's building a list of all files in
memory.  Anyway, I recommend replacing cp with rsync, even if the
--bwlimit option isn't used.  I usually use rsync for local-local
copies of anything over 10 MB; it's smarter in many ways.  On my
machine, rsync only uses 4MB of virtual memory (2MB resident
memory) regardless of file sizes or number of files.

Also, if the backup script is using 'cp', that suggests it isn't
using the "--link-dest" option available in newer versions of
rsync.


>>>>> On 2005-12-23 13:20 PST, Ryan M writes:

    Ryan> There must be a that is too large, even for swap.  While
    Ryan> cp doesn't put everything into memory that it copies,
    Ryan> linux does store a good deal of what it accesses into
    Ryan> cache.  Depending on how the Linux vm proceedes onces
    Ryan> the low memory limit (on 2.6. kernels ONLY,
    Ryan> /proc/sys/vm/min_free_kbytes) is reached, that could
    Ryan> spill over into swap or the cache could be relieved as
    Ryan> cp continues its job and therefore physical memory not
    Ryan> ever running out. There IS a min_free_kbytes somewhere
    Ryan> in the 2.4 kernel -- there has to be, but I have no idea
    Ryan> _how_ the vm uses it to make decisions about what to do
    Ryan> with overages in memory use.  My vm in 2.6 very well
    Ryan> behaved, that's all I can say.  Copies never touch swap,
    Ryan> but do drain physical memory by over 300mb when the
    Ryan> operation is on an unclean kernel tree (345mb large).

    Ryan> Take a look at the output of /proc/meminfo (maybe even
    Ryan> save it) over a period of time during the backup
    Ryan> process.  Or look at top, but that isn't a very thorough
    Ryan> representation of what is going on.  Note that we have,
    Ryan> at the moment, only 34mb of physical memory available.
    Ryan> In order to do what the kernel desires, it should spill
    Ryan> over into swap.  But taking 3gb in a single copy and the
    Ryan> vm eating every bit of swap... that can't be? :-) Or is
    Ryan> it complaining that it cannot read x into memory because
    Ryan> it is larger than the 34mb available?  How does the 2.4
    Ryan> vm work?

    Ryan> I'm guessing that recently there has been more disk use
    Ryan> wrt space taken?  Can someone test my theory?  Can we
    Ryan> split the cp operations?  Once the cp is finished, and a
    Ryan> new process takes over, the cached / swapped data should
    Ryan> be purged.  IF this is the case, we can have it count
    Ryan> how much it has already read, and stop at a certain
    Ryan> amount (say, 300mb), then start over again from where it
    Ryan> left off.


    Ryan> Ryan


    Ryan> On Fri, 23 Dec 2005 08:23:40 -0800, AC wrote about
    Ryan> "[Hcoop-discuss] rsnapshot errors" :

    >> Lately, the rsnapshot backup program has been crashing like
    >> this, so we don't actually have any especially recent
    >> backups.  Does anyone have any idea how we could get around
    >> the error, which I've included below?
    >> 
    >> /bin/cp: memory
    >> exhausted ----------------------------------------------------------------------------
    >> 
    >> rsnapshot encountered an error! The program was invoked
    >> with these options: /usr/bin/rsnapshot
    >> daily ----------------------------------------------------------------------------
    >> 
    >> ERROR: /bin/cp failed. Perhaps this is not GNU cp?  ERROR:
    >> Error!  cp_al("/home/backup/rsnapshot/daily.0/",
    >> "/home/backup/rsnapshot/daily.1/")
    >> 
    >> _______________________________________________
    >> Hcoop-discuss mailing list Hcoop-discuss at hcoop.net
    >> http://hcoop.net/cgi-bin/mailman/listinfo/hcoop-discuss

    Ryan> _______________________________________________
    Ryan> Hcoop-discuss mailing list Hcoop-discuss at hcoop.net
    Ryan> http://hcoop.net/cgi-bin/mailman/listinfo/hcoop-discuss

-- 
Karl 2005-12-23 18:35