Dear Friends
Can anyone help me write a script for incremental backup using rsync to create multiple restore-point-backup that can be easily restored?
I feel a full backup once in 400 days, with hourly incremental, daily incremental and weekly incremental backup being rotated in a way in which 1 full backup followed by 24 hourly incremental backups, interspersed with 7 daily incremental backups and 5 weekly incremental backups may be protection enough for my data. What is the you all's expert opinion? Kindly suggest a better methodology please.
Would it be better if these backups are compressed and stored in tar format? Kindly guide please.
Regards and thank you all in advance Kshitiz
2010/11/22 Kshitiz kshitij_kotak@hotmail.com:
I feel a full backup once in 400 days, with hourly incremental, daily incremental and weekly incremental backup being rotated in a way in which 1 full backup followed by 24 hourly incremental backups, interspersed with 7 daily incremental backups and 5 weekly incremental backups may be protection enough for my data. What is the you all's expert opinion?
I am assuming you meant 40 days, and not 400 days. :-)
Let us say a full backup takes 30 minutes to restore, each weekly incremental takes 5 minutes to apply and each daily incremental, 1 minute. Let us further say you have an outage 33 days after your previous full backup. Now, the time for you to restore is:
30 minutes to restore the last full backup 4 x 5 = 20 minutes to apply the 4 weekly incrementals since the last full backup 5 x 1 = 5 minutes to apply the 5 daily incrementals since the last weekly incremental
Total time taken to restore = 55 minutes ~= 1 hour.
Now, nobody else apart from you can say whether 1 hour of downtime while you restore from backups is acceptable or not. The right way to design your backup schedule is to first determine what the acceptable limit on this time taken to restore (called MTTR, or Mean Time To Recover - a little bit of statistics enter here to determine this "mean") is, and then work backwards. You will find that your entire backup strategy can change with your acceptable MTTR.
A second issue you haven't addressed is that the smallest incremental you have is hourly. So then what happens if an outage occurs 55 minutes after the last hourly incremental? Are you fine with losing 55 minutes worth of data?
Binand
On Mon, Nov 22, 2010 at 7:43 AM, Binand Sethumadhavan binand@gmail.com wrote:
2010/11/22 Kshitiz kshitij_kotak@hotmail.com:
I feel a full backup once in 400 days, with hourly incremental, daily incremental and weekly incremental backup being rotated in a way in which 1 full backup followed by 24 hourly incremental backups, interspersed with 7 daily incremental backups and 5 weekly incremental backups may be protection enough for my data. What is the you all's expert opinion?
I am assuming you meant 40 days, and not 400 days. :-)
Let us say a full backup takes 30 minutes to restore, each weekly incremental takes 5 minutes to apply and each daily incremental, 1 minute. Let us further say you have an outage 33 days after your previous full backup. Now, the time for you to restore is:
30 minutes to restore the last full backup 4 x 5 = 20 minutes to apply the 4 weekly incrementals since the last full backup 5 x 1 = 5 minutes to apply the 5 daily incrementals since the last weekly incremental
Total time taken to restore = 55 minutes ~= 1 hour.
Nice analysis.
I feel we can provide better insight if he lets us know what exactly he is trying to backup. Since he is using rsync, I assume he is backing up files at the OS level. Now, what application does he use, and how often to files get generated/modified at the OS level? That would decide the frequency of his full and incremental backups.
If we are taking about OLTP here, then logic says the data be stored in a database, not at the OS level. In such a case, the db will take care of mttr, etc. The 55 minutes of data loss would be taken care of by the redo logs (if Oracle).
I have never come across a situation where I had to run hourly incremental backups at the OS level.
Regards, NMK.
-------------------------------------------------- From: "Nadeem M. Khan" nadeem.m.khan@gmail.com Sent: Monday, November 22, 2010 10:13 AM To: "GNU/Linux Users Group, Mumbai, India" linuxers@mm.ilug-bom.org.in Subject: Re: [ILUG-BOM] Incremental Backup using rsync
On Mon, Nov 22, 2010 at 7:43 AM, Binand Sethumadhavan binand@gmail.com wrote:
2010/11/22 Kshitiz kshitij_kotak@hotmail.com:
I feel a full backup once in 400 days, with hourly incremental, daily incremental and weekly incremental backup being rotated in a way in which 1 full backup followed by 24 hourly incremental backups, interspersed with 7 daily incremental backups and 5 weekly incremental backups may be protection enough for my data. What is the you all's expert opinion?
I am assuming you meant 40 days, and not 400 days. :-)
Let us say a full backup takes 30 minutes to restore, each weekly incremental takes 5 minutes to apply and each daily incremental, 1 minute. Let us further say you have an outage 33 days after your previous full backup. Now, the time for you to restore is:
30 minutes to restore the last full backup 4 x 5 = 20 minutes to apply the 4 weekly incrementals since the last full backup 5 x 1 = 5 minutes to apply the 5 daily incrementals since the last weekly incremental
Total time taken to restore = 55 minutes ~= 1 hour.
Nice analysis.
I feel we can provide better insight if he lets us know what exactly he is trying to backup. Since he is using rsync, I assume he is backing up files at the OS level. Now, what application does he use, and how often to files get generated/modified at the OS level? That would decide the frequency of his full and incremental backups.
If we are taking about OLTP here, then logic says the data be stored in a database, not at the OS level. In such a case, the db will take care of mttr, etc. The 55 minutes of data loss would be taken care of by the redo logs (if Oracle).
I have never come across a situation where I had to run hourly incremental backups at the OS level.
Regards, NMK. -- http://mm.glug-bom.org/mailman/listinfo/linuxers
Thank you Nadeem & Binand for your reply. Will revert to you asap.
Regards Kshitiz
On Mon, 22 Nov 2010, Kshitiz wrote:
Dear Friends
Can anyone help me write a script for incremental backup using rsync to create multiple restore-point-backup that can be easily restored?
I feel a full backup once in 400 days, with hourly incremental, daily incremental and weekly incremental backup being rotated in a way in which 1 full backup followed by 24 hourly incremental backups, interspersed with 7 daily incremental backups and 5 weekly incremental backups may be protection enough for my data. What is the you all's expert opinion? Kindly suggest a better methodology please.
Would it be better if these backups are compressed and stored in tar format? Kindly guide please.
Regards and thank you all in advance Kshitiz
Why reinvent the wheel ? Look at rsnapshot. Been using it for a couple of years now.
regards,
Sharukh.
-------------------------------------------------- From: "Dr. Sharukh K. R. Pavri." lists@pavri.net Sent: Monday, November 22, 2010 4:33 PM To: linuxers@mm.ilug-bom.org.in Subject: Re: [ILUG-BOM] Incremental Backup using rsync
On Mon, 22 Nov 2010, Kshitiz wrote:
Dear Friends
Can anyone help me write a script for incremental backup using rsync to create multiple restore-point-backup that can be easily restored?
I feel a full backup once in 400 days, with hourly incremental, daily incremental and weekly incremental backup being rotated in a way in which 1 full backup followed by 24 hourly incremental backups, interspersed with 7 daily incremental backups and 5 weekly incremental backups may be protection enough for my data. What is the you all's expert opinion? Kindly suggest a better methodology please.
Would it be better if these backups are compressed and stored in tar format? Kindly guide please.
Regards and thank you all in advance Kshitiz
Why reinvent the wheel ? Look at rsnapshot. Been using it for a couple of years now.
regards,
Sharukh.
Dr. Sharukh K. R. Pavri. Homoeopath, Linuxer. It's faster horses, Younger women, Older whiskey and More money. -- Tom T. Hall, "The Secret of Life" -- http://mm.glug-bom.org/mailman/listinfo/linuxers
Thank you Dr. Will try and configure it and revert in case of any help / support required.
Regards Kshitiz
On Monday 22 Nov 2010, Dr. Sharukh K. R. Pavri. wrote:
Dr. Sharukh K. R. Pavri. Homoeopath, Linuxer. It's faster horses, Younger women, Older whiskey and More money. --
If that's the homeopathic formula for living a longer and healthier life, I'm a convert!
-- Raj
On Sun, 28 Nov 2010, Raj Mathur (????????? ???????????????) wrote:
On Monday 22 Nov 2010, Dr. Sharukh K. R. Pavri. wrote:
Dr. Sharukh K. R. Pavri. Homoeopath, Linuxer. It's faster horses, Younger women, Older whiskey and More money. --
If that's the homeopathic formula for living a longer and healthier life, I'm a convert!
-- Raj
It's a universal formula. No need to convert.
Sharukh.
On Mon, Nov 22, 2010 at 12:18 AM, Kshitiz kshitij_kotak@hotmail.com wrote:
Dear Friends
Can anyone help me write a script for incremental backup using rsync to create multiple restore-point-backup that can be easily restored?
I feel a full backup once in 400 days, with hourly incremental, daily incremental and weekly incremental backup being rotated in a way in which 1 full backup followed by 24 hourly incremental backups, interspersed with 7 daily incremental backups and 5 weekly incremental backups may be protection enough for my data. What is the you all's expert opinion? Kindly suggest a better methodology please.
Would it be better if these backups are compressed and stored in tar format? Kindly guide please.
Try this http://backuppc.sourceforge.net/
Very simple to deploy and use with web based control panel
Regards and thank you all in advance Kshitiz
HTH With regards,
-
Binand Sethumadhavan -
Dinesh Shah (દિનેશ શાહ/दिनेश शाह) -
Dr. Sharukh K. R. Pavri. -
Kshitiz -
Nadeem M. Khan -
Raj Mathur (राज माथुर)