Not all volumes get backed up

gary.braman's Avatar

gary.braman

04 Sep, 2013 02:19 PM

I created a scheduled task to create a snapshot of all volumes nightly. Only 109 of the 267 volumes in my account are getting backed up.

I've tried the wildcard * tag and using a keyword (VOLSNAP) on each volume. Is there a limit to the number of snapshots that can be requested at one time? I'd really like to be able to just use the wildcard operator so I don't have to edit tags every time I create a volume.

  1. 1 Posted by simon on 20 Sep, 2013 02:19 PM

    simon's Avatar

    I am having the same experience. Backup Tasks failing:
    "Backup All Instances" with tags containing "01" "02" "03" or "04" for different groups of systems are failing.

    EC2 instances are in VPC. Ireland Region.

    The tasks backup about 50 instances and only 20 are being backed up. There seems to be little pattern to the behaviour, apart a lower purge period seems to cause more failures. I am going to try increasing the purge period to a crazy number tonight and see if this helps...

    These tasks were working until 10 days ago, did somehting change on 10th or 11th Sept?

    Rgds, Simon

  2. Support Staff 2 Posted by Ylastic on 26 Sep, 2013 12:28 PM

    Ylastic's Avatar

    No changes on our end. AWS on their end limits the number of snapshots that can be in progress in each region at any given time.

  3. 3 Posted by Phil on 05 Jan, 2014 11:40 AM

    Phil's Avatar

    "AWS on their end limits the number of snapshots that can be in progress in each region at any given time."

    I'm not seeing any way to schedule a particular time for snapshots. Why is that?

    So given this particular AWS behavior, what is Ylastic planning to do about it? Throwing up your hands and blaming Amazon isn't solving the problem. Two simple solutions that come to mind are: 1) stagger the snapshot schedule so that only a few, or 1, are being scheduled at a given time. 2) Automatically retry failed snapshot attempts (up to N times) after a certain time.

  4. 4 Posted by gary.braman on 05 Jan, 2014 11:56 AM

    gary.braman's Avatar

    I will be out of the office until Tuesday, January 7, 2014 with limited access to email. If you need immediate assistance, please call the help desk at 480.346.1299.

    Thanks.

  5. Support Staff 5 Posted by Ylastic on 05 Jan, 2014 04:00 PM

    Ylastic's Avatar

    You can set a specific time for snapshots using the cron option. You specify a cron string for the exact time.

    We changed things so we can handle any number of volumes to be snapshot, and we handle the too many snapshots issue as well. Could you please try your task again? One thing to keep in mind. If you have a lot of volumes, it will take a little bit of time to make snapshots of them. This happens in the background.

    thanks!

  6. 6 Posted by gary.braman on 05 Jan, 2014 04:56 PM

    gary.braman's Avatar

    I respectively disagree; the scheduled snapshot solution just doesn’t work as you describe. I am trying to backup over 300 volumes. . I had successful snapshots running on schedules using Ylastic up until Dec 13, 2013. At this point, the Ylastic scheduled snapshots just stopped working. I’ve tried breaking my TAGS down into groups and have scheduled many different variations of TAGs and CRON and other schedules over that past month trying to fix this issue with limited luck. Even when the task status shows as successful, not all volumes get a snapshot completed. Occasionally, I do get a message that the task failed doe to the AWS snap shot limit being reached. I really need this feature to work. I’m actively searching for an alternate solution at a reasonable cost as I just can’t depend on Ylastic at this point. Support replies are extremely slow as well.

  7. Support Staff 7 Posted by Ylastic on 05 Jan, 2014 05:00 PM

    Ylastic's Avatar

    This morning I ran a task to snapshot 150 volumes in us east region. The task used cron and tags. It ran correctly and created 150 snapshots. Sorry about the issue you are having with this feature. Would you like to try running this task one more time?

    thanks!

  8. 8 Posted by gary.braman on 05 Jan, 2014 05:04 PM

    gary.braman's Avatar

    Can you please share the parameters you are using on your snapshot task?

    Action, Tag value, snapshots to keep, and (every, at, or CRON settings?)

    Thanks.

  9. Support Staff 9 Posted by Ylastic on 05 Jan, 2014 05:09 PM

    Ylastic's Avatar

    This task runs every 12 hours.

  10. 10 Posted by gary.braman on 05 Jan, 2014 05:16 PM

    gary.braman's Avatar

    Thanks for the screenshot.

    I’ll try to run another bulk snapshot job.

    I’d really like to just have one wildcard CRON snapshot job running for each region. I don’t care how long it takes (within reason) to snapshot all drives.

    [cid:[email blocked]]

    One thing that I noticed that may be affecting my snapshots is when I click on the backups link under the monitor menu, I see a good number of backups that have a status of In Progress or even a status of created with a yellow “in progress” dot with creation dates as far back as 11-28-2013 I deleted all of my scheduled tasks to run scheduled backups (AMI)

    [cid:[email blocked]]

  11. Support Staff 11 Posted by Ylastic on 05 Jan, 2014 06:23 PM

    Ylastic's Avatar

    I think we are talking about two different things. The first screenshot showing backup AMI is a different task (that is the task for backing up instances to AMI). The backup volumes is a different task. It is for backing up volumes.

    I will test the backup instances to AMI task too to see what the issue maybe.

    You should be able to run one backup volume job for each region. If it does not work correctly, I will ensure that we fix it so it does work like that. That is the way we want things to be, as easy as one task to backup all volumes in each region while keeping the snaps pruned on a schedule.

    thanks!

  12. 12 Posted by gary.braman on 05 Jan, 2014 07:02 PM

    gary.braman's Avatar

    Sorry for the confusion. I realize that the AMI backup is different than the snapshots, but because of the confusing status of the AMI backup (which creates their own snapshots), I thought perhaps they were impeding the scheduled Snapshsots.

    I’ve deleted all of my scheduled task and have created a wild card task to snapshot all volumes in Oregon. At this point, I am seeing snapshots being created and old snapshots being deleted as expected. If this task completes successfully, it won’t be for many hours. I’ll keep you posted. Thanks so much for your responses today. I’ve been fighting this for several weeks since my original snapshot schedule started failing.

    [cid:[email blocked]]

  13. Support Staff 13 Posted by Ylastic on 05 Jan, 2014 07:10 PM

    Ylastic's Avatar

    No worries. If you have any issues at all with it, please let me know.

    thanks for your patience!

  14. 14 Posted by gary.braman on 06 Jan, 2014 07:43 PM

    gary.braman's Avatar

    I recreated the task as a wild card and a snapshot was created for all but 12 drives over a period of 20+ hours. I did make a change to the scheduled task to change the start time at about the 20 hour mark. I’m wondering if making this change stopped any task that might have been running and in turn kept the last 12 snapshots from being created. I’ll know in a 36 hours or so. Feeling optimistic. Thanks.

  15. Ylastic closed this discussion on 08 Aug, 2014 02:18 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac