RDS backup multiple failures

Roman Belov's Avatar

Roman Belov

07 Jun, 2015 03:30 AM

Hi there,

Lately we started to see following backup errors:
Scheduled snapshot for RDS instance failed for task [rds xxxxxxxxx snapshot] in account [yyyy] - Rate exceeded.
Today I've got more tnan 10 errors like this for various RDS instanses.

Please, advice

Thanks,
Roman

  1. Support Staff 1 Posted by Ylastic on 07 Jun, 2015 12:06 PM

    Ylastic's Avatar

    Hi Roman,

    Do you have any other AWS Api calls being made around the time this task ran?

    thanks

  2. 2 Posted by Roman Belov on 07 Jun, 2015 01:05 PM

    Roman Belov's Avatar

    It is possible that we have number of API Calls at the same time - we have relatively big system which starts hundreds of spot instances on our appication demand snd shuts them down when they finish their job, etc.
    We also have around more than 40 DB's and probably more on-demand EC2 instances and webackup all of them through Ylastic tasks so is possible that number of such backup requests coming together or within short time interval.
    Is there any existing limit for AWS API calls?

    Thanks,
    Roman.

  3. Support Staff 3 Posted by Ylastic on 07 Jun, 2015 01:56 PM

    Ylastic's Avatar

    All AWS API calls are rate limited. We added a workaround to delay the describe calls by 2 seconds. Next time the task runs, the change should be in effect.

  4. 4 Posted by Roman Belov on 08 Jun, 2015 05:50 AM

    Roman Belov's Avatar

    Still having same issue.
    Looks like this 2sec delay does not helps.
    Is there any specific limit which I could request AWS support to be increased?

  5. Support Staff 5 Posted by Ylastic on 08 Jun, 2015 01:19 PM

    Ylastic's Avatar

    There is API call limit that AWS enforces. Maybe it would be possible to raise that limit for your AWS account? If you want, we can also experiment with increasing the delay and staggering it a bit more on our side as a workaround.

  6. 6 Posted by Roman Belov on 10 Jun, 2015 04:07 PM

    Roman Belov's Avatar

    The thing is that all RDS backups actually being completed successfully.
    So there is nothing to take care of on AWS side.
    I only see these "Rate Limit" errors in Ylastic, which seem to be unable to recognize that backup request has been accepted and processed.
    Please, take another look into it.

  7. Support Staff 7 Posted by Ylastic on 10 Jun, 2015 04:10 PM

    Ylastic's Avatar

    Will look into this more and see if we can find out why this is happening.

    thanks for the debugging info!

  8. 8 Posted by Roman Belov on 18 Jun, 2015 08:48 AM

    Roman Belov's Avatar

    Can it be that the root cause is big number of DB instances which should get backed up on single task run?
    We have number of tasks which regex applies to 10 different DB's.

  9. Support Staff 9 Posted by Ylastic on 18 Jun, 2015 12:52 PM

    Ylastic's Avatar

    That is probably the reason. The issue is not when making the call for the DB snapshot, but making API calls after that to make sure the process finished that are having this issue. We have a new version deployed just now. Next time the task runs, please let me know if you get the error.

    thanks

  10. 10 Posted by Roman Belov on 29 Jun, 2015 10:04 AM

    Roman Belov's Avatar

    I had never seen these errors since your notification about new version deployed, so it seems to be fixed now.

    Thanks,
    Roman.

  11. Support Staff 11 Posted by Ylastic on 29 Jun, 2015 10:18 AM

    Ylastic's Avatar

    Great. Closing this issue.

    We have received several requests for a new scheduled task for copying RDS snapshots between regions on a schedule. Just curious if this is something that you guys would be interested in, as you already do the RDS backups.

    thanks

  12. Ylastic closed this discussion on 03 Feb, 2017 01:33 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac