Nature of backups changed

Garret Cox's Avatar

Garret Cox

17 Dec, 2013 08:39 AM

I have a series of schedule tasks set to "Backup All EBS Instances", that I use to create nightly AMIs.
Unfortunately, as the nature of the servers don't lean well towards nightly reboots, I had the tasks all set to NOT reboot the instance to ensure file consistency.

Around Saturday at 2:00 central (the time of my scheduled backups) ALL of my servers rebooted. It's happened again each night since. It took me until tonight to realize that this was being caused by Ylastic: upon inspecting the scheduled tasks, I discovered that now the checkbox to reboot the servers had been enabled!

I'm just wondering if anyone else has had this problem, or if Ylastic has an explanation for the change...because if not, I've got to find out if anyone else is screwing around with my Ylastic configuration.

  1. Support Staff 1 Posted by Ylastic on 17 Dec, 2013 01:32 PM

    Ylastic's Avatar

    Garrett,

    The backups task had a bug associated with the flag for reboot. When you created the task, you had the flag enabled, but that flag was not getting passed to the API call. Since we fixed the issue the flag is getting passed. Apologies for any inconvenience caused due to this.

    thanks

  2. 2 Posted by Garret Cox on 17 Dec, 2013 04:35 PM

    Garret Cox's Avatar

    Thanks for the response, I'm just glad that:
    1. The issue is found and resolved
    2. it's not someone unauthorized messing with my backup settings.

    Merry Christmas!

  3. 3 Posted by Brandon Adams on 18 Dec, 2013 07:09 PM

    Brandon Adams's Avatar

    Are you certain that's the whole extent of the bug fix?

    I found that all but two of our schedule instance backups now had this flag enabled, even though I always choose to disable it. An instance backup task I created just last week now had it enabled, and I distinctly remember disabling it then.

    I'm wondering if maybe the UI wasn't submitting the option correctly in the past, too.

    8 of my tasks now had the reboot flag checked, only 2 had it unchecked.

  4. Support Staff 4 Posted by Ylastic on 18 Dec, 2013 07:37 PM

    Ylastic's Avatar

    @brandon the bug should be fixed now. If you edit and save the task is it not saving your change?

    thanks!

  5. 5 Posted by Brandon Adams on 18 Dec, 2013 07:44 PM

    Brandon Adams's Avatar

    It does save, I just wanted to confirm that there was another component of this bug. That is, the two parts were:

    1. When checked, the option wasn't actually applied at task run time.
    2. When unchecked and saved, the option was recorded as checked.

    That would explain the behavior we saw.

    Also, is there any kind of feed I can subscribe to for things like this? This cost us a couple hours of labor before we discovered the root cause.

  6. Support Staff 6 Posted by Ylastic on 18 Dec, 2013 08:04 PM

    Ylastic's Avatar

    Yes, you are correct.

    I am sorry we dont have any feed for this. I apologize for the inconvenience. We are going to do better at this and keeping you informed of bug fixes in addition to the normal release stuff.

    thanks for your patience!

  7. 7 Posted by Roman Belov on 19 Dec, 2013 08:06 AM

    Roman Belov's Avatar

    Few more questions regarding this issue:
    1. Why do you use Control-Alt-Delete for such a shutdown even it is requested?
    Right after I discovered the reason of my server reboots, I had set them to ignore Control-Alt-Delete requests. Does it means that now this option will never work until I will re-enable it?
    2. When you perform such a reboot - why it comes from root@unknown???
    This is why it took me and probably others, just like Brandon wrote, hours of labor to figure out the root cause.
    3. I would also connect to some kind of feed informing me about such bug fixes or otther issues which could affect operation of my production servers - in other way I will not feel myself safe enough using Ylastic.
    Even if it is not exist now, you should create it or at least, send mail notification about such changes or fixes to all your customers, list of which you surely have.

  8. Support Staff 8 Posted by Ylastic on 19 Dec, 2013 02:38 PM

    Ylastic's Avatar
    1. We are not using control-alt-delete. AWS does that when a shutdown is requested. All we do is pass the enabled or disabled flag to aws via the API.

    2. Once again, we do NOT control how AWS does the reboots. There is no parameter passed by us to the API that indicates root@unknown.

    3. We will add this to email notifications.

    thanks!

  9. Ylastic closed this discussion on 08 Aug, 2014 02:40 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac