Scheduled task for volumes doesn't fully work sometimes

ender.gorgulu's Avatar

ender.gorgulu

27 Nov, 2013 08:18 AM

Hi,

I have a scheduled task which takes a snapshot every night at 01:00 (CET+6) on 4 volumes. It only keeps the last one every time.
It's been working without any problems until today and I think I have noticed the problem earlier but did not think about it so much.

Instead of removing the snapshot from yesterday which it used to do and then take a new one, I know 4 snapshots from 2013-11-26 01:00 GMT+0100 and 4 new snapshots from 2013-11-27 01:00 GMT+0100.
Earlier it always kept the last snapshots and removed the other ones.

Is this a known problem or am I doing something wrong?

My task looks like this:

Task: "Snapshot all EBS Volumes whose name tag contains [SALL-SQL] keeping only latest 1 snapshots in Ireland"
Schedule: cron [0 19 * * *]
Last Run: Tue Nov 26 19:00:00 -0500 2013
Last Status: Successful

  1. Support Staff 1 Posted by Ylastic on 01 Dec, 2013 04:00 PM

    Ylastic's Avatar

    Could you please clarify this? Are you saying the task runs but occassionally leaves extra snapshots?

    thanks

  2. 2 Posted by ender.gorgulu on 02 Dec, 2013 08:56 AM

    ender.gorgulu's Avatar

    Hi,

    I have one scheduled snapshot task which is set to take snapshots on all volumes with the tag CU-SALL-SQL everyday and only keep the latest one.

    I will try to explain again:
    When I started the scheduled task, the first week it worked as intended. Everytime a snapshot was taken the older one was removed so I always had 4 snapshots (1snapshot for each volume) every day.

    After a week or so I noticed that I had 8 snapshots in my inventory instead of 4. When I looked at the dates I noticed that the task had created 4 new snapshots and left the older ones without deleting them. So I manually deleted the old snapshots that day.

    Next day same thing happened. I had once again 8 snapshots in the inventory. I deleted the old ones and let the new ones stay....but the problems keeps coming back.

    As a last thing I deleted the task and all the snapshots but after a couple of days the problem kept coming back.

    So now I haveto delete the old snapshots manually since the task doesn't seem to take care of them anymore.

  3. 3 Posted by RichBos on 10 Dec, 2013 06:35 PM

    RichBos's Avatar

    +1 for this, we have this same issue although on a much larger scale, my client has multiple instances and has had no snapshot prune for 3 months, as such he has ended up with almost 1500 snapshots....! It's an absolute nightmare.

    The snapshots take but don't prune, and now we can't edit or recreate the tasks as the system says there are no volumes in the region we are using...! The whole task/snapshot/prune seems to have totally fallen apart.

  4. 4 Posted by ender.gorgulu on 12 Dec, 2013 10:43 AM

    ender.gorgulu's Avatar

    I'm sorry but my trust for this product is gone.

    To wait several days just to get an answer for a support case is not sustainable.
    This "bug" costs alot of money if you don't keep an eye on it since snapshots aren't free and this task can grow for ages.
    Like RichBos said, 1500snapshots is ALOT of money and a nightmare if you don't notice it in time.

    I'm sorry but I think I haveto look for another solution.

  5. Support Staff 5 Posted by Ylastic on 12 Dec, 2013 01:53 PM

    Ylastic's Avatar

    @RichBos Are you using two tasks - one to create snaps, and one to prune snaps? If you have tags on the volume, you can use the single task that snaps and prunes. Which one are you using?

  6. 6 Posted by RichBos on 12 Dec, 2013 02:51 PM

    RichBos's Avatar

    Hi, we are using two tasks, however even if a single task using tags works and is more efficient the fact remains that the console is reporting all tasks (which it initially allowed us to configure) with success, which is incorrect and led us to believe everything was ok.
    We trusted the console advisories but are now wondering what other faults there could be elsewhere in the system and what else may be being reported incorrectly? How can we know for sure anymore?

    Apologies but we would need a serious restoration of faith to continue using Ylastic in production.

  7. Support Staff 7 Posted by Ylastic on 12 Dec, 2013 03:15 PM

    Ylastic's Avatar

    @ender.gorgulu Could you please create your task again, so we can check it and ensure that this does not happen again?

    t

  8. Support Staff 8 Posted by Ylastic on 12 Dec, 2013 03:21 PM

    Ylastic's Avatar

    @RichBos Would you like to try out the single task instead of the two separate ones. We should have caught this issue with the display reporting incorrectly. Sorry about that. We would love to continue to have you guys using Ylastic.

  9. 9 Posted by RichBos on 12 Dec, 2013 03:38 PM

    RichBos's Avatar

    The system will not allow us to create a new snapshot task as it says we have no volumes in the Sydney region. We do, obviously (See my previous comment, No.4 in this thread), second paragraph.

  10. Support Staff 10 Posted by Ylastic on 12 Dec, 2013 06:44 PM

    Ylastic's Avatar

    That message is displayed when you use the task that only creates snapshots. The default option when you create a new task is 'Snapshot all EBS volumes' which uses tags and also prunes. Is that the new task you are trying to setup?

  11. Support Staff 11 Posted by Ylastic on 12 Dec, 2013 06:47 PM

    Ylastic's Avatar

    attaching a screenshot

  12. 12 Posted by RichBos on 13 Dec, 2013 10:30 AM

    RichBos's Avatar

    This is not a solution. The system won't let us apply tags anyway, so that is another issue.

    The question remains for us as to why the tasks we have set up are showing as successful and why this happened in the first place? We are very interested to hear your reply with regard to the cost implications this created for the client.

    As things are the client has lost confidence in the service and is unwilling to take further risk, no matter how many 'other' solutions or 'workarounds' you advise. The bottom line is that we set up snapshot and prune tasks in a logical way, the system allowed us to do so and reported success. If the prune task was failing we should have been notified, period.

    Apologies but we cannot take another chance with this, it was too big a problem. We have found a simpler, cheaper option for snapshots and will be disabling all Ylastic tasks over the coming week.

  13. 13 Posted by RichBos on 13 Dec, 2013 10:31 AM

    RichBos's Avatar

    For reference here is a grab showing the tasks as reported.

  14. 14 Posted by ender.gorgulu on 13 Dec, 2013 10:34 AM

    ender.gorgulu's Avatar

    RichBos, could u contact me on [email blocked]. I have a couple of questions and I can't seem to send private messages from here :(

    Thanks!

  15. 15 Posted by RichBos on 15 Dec, 2013 08:29 AM

    RichBos's Avatar

    I see the thread has dried up since the mention of cost implications, shame, a degree of responsibility for failure of the product would have been preferred.

  16. Ylastic closed this discussion on 08 Aug, 2014 02:18 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac