Stream proxied S3 and Azure artifacts - #8085
Conversation
2f385d4 to
7d77c41
Compare
Bypass django-storages eager file buffering when the content app proxies S3 or Azure artifacts. Preserve the generic storage fallback and HTTP range behavior. closes pulp#7806 Assisted By: Codex (GPT-5) Terra 5.6
7d77c41 to
2ed7659
Compare
| @@ -0,0 +1,166 @@ | |||
| """Temporary private streaming adapters for object-storage response bodies. | |||
There was a problem hiding this comment.
Ideally none of these wrappers are necessary, and django-storages would allow us to do something like this natively: jschneier/django-storages#1568
But for now, we cannot, and the buffering implementation is genuinely unsuitable for sufficiently large files (context: a user has reported that they are storing files ranging up to 60 or even 300 gigabytes)
There was a problem hiding this comment.
Obviously this is relatively invasive, so we need to be very careful with testing.
There was a problem hiding this comment.
Alternatively, we could temporarily fork django-storages and add better APIs directly to the implementation. It is not in a great place maintenance-wise anyway currently.
There was a problem hiding this comment.
Here is what the upstream PR looks like jschneier/django-storages#1569
|
@mdellweg I'm marking this draft because it probably needs more extensive testing, however, I would still like feedback on the direction and approach. e.g. if you want to try the "vendor/fork django-storages" approach. Both the issue itself and the upstream status of the library is a real issue we're going to need to deal with one way or the other. |
|
First, I don't know what the best approach is. Each approach has pro's and con's. |
Bypass django-storages eager file buffering when the content app proxies S3 or Azure artifacts. Preserve the generic storage fallback and HTTP range behavior.
closes #7806
Assisted By: Codex (GPT-5) Terra 5.6
📜 Checklist
See: Pull Request Walkthrough