Repository navigation
Missing Limits module #72
Description
Activity
@kevinhughes27 Beyond just porting this limiting module over, do you think it would be feasible to somehow build rate limit handling transparently into the API client?
What I mean by that is having all API calls handle a rate limit exceeded exception and automatically retry (following the directive in the returned
Retry-Afterheader). Then you could do something like:for product in products: product.add_metafield(...)and not have to worry about catching a rate limit exception and manually restarting your requests.
My initial thought would be that this might require too much digging into pyactiveresource's internals, just wanted to float the idea.
Reacted by Volodymyr, desiredState and Cole Jancsar@gavinballard its a cool idea and something we discussed doing before. I think you are right - it would require too much digging into pyactiveresource internals. I think a better solution would be a library that sits above your api client and manages this. There is a ruby gem somewhere that sort of does this but I can't find it at the moment.
Fair enough, I think you're right :). If I come up with a good pattern for this I'll share here.
@kevinhughes27 Hey Kevin, don't suppose you ever tracked down that Ruby gem doing something along these lines?
maybe this one? https://github.com/ejfinneran/ratelimit we still don't have a good solution for this
Us either :). We're looking to build out a solution on the Ruby side of things, might try to port over to Python if that works out.
deffs let us know about it!
@raulbrito, who's looking into this with me, found this article which is quite relevant: http://product.reverb.com/2015/03/07/shopify-rate-limits-sidekiq-and-you/.
Not a bad approach at all!
interesting. Thanks for sharing! Api limiting might be better built at the app framework level like the shopify_app gem especially if it needs to connect to the background queue like this
FYI, I wrote a HTTP proxy that can rate limit outbound HTTP calls - http://github.com/mrkschan/cuttle. You may also find the Shopify API setup at - http://mrkschan.blogspot.hk/2015/10/rate-limiting-shopify-api-using-cuttle.html.
@mrkschan: Thanks for sharing a great approach!
I've solved this in my projects by implementing a token bucket algorithm that keeps track of recent requests within a Redis server, per account. The token bucket works well with Shopify's leaky bucket implementation- the leaky bucket starts at zero, gets added to, and overflows are ignored (throw an error), while the token bucket starts with a number of tokens which are then consumed, and when there are no tokens left to consume, there is essentially a queue waiting to consume them. I've monkey-patched the ShopifyConnection class to consume or wait before sending out the request, and it works great. This is the only way I've come across that utilizes true "bursts"- if I send 50 requests simultaneously, the first 40 will get sent right away while the remaining 10 get sent every 0.5 seconds.
very cool! @flux627 is your solution available anywhere? Its not the kind of thing we would include here since it would introduce a dependency on redis but it would be worth linking to.
Thanks!
is this happening?
So, is there any native way to handle
Exceeded 4 calls per second for api clienterror now?
Or ideas where to implement it as part of this library?Here is my solution to this problem, a code will just wait some time and will retry request:
https://gist.github.com/wowkin2/079844c867a1a06ce15ea1e4ffdee87cReacted by Sameer Kumar and Andreas ScheucherThanks for contributing! This package is deprecated and we're archiving this repo, so we're closing all open issues and PRs. Please use shopify-app-python instead, and open new issues there. More context: Rethinking support for PHP/Python packages
The ruby client has:
https://github.com/Shopify/shopify_api/blob/master/lib/shopify_api/limits.rb
which has not been ported into the python lib