In a quick read about the use_accelerate_endpoint, it sounds like it only has benefit when the client is in a different region from the bucket and the files are largish. Should there be any sort of check for either of these conditions, since using use_accelerate_endpoint incurs higher transfer costs?

In a quick read about the use_accelerate_endpoint, it sounds like it only has benefit when the client is in a different region from the bucket and the files are largish. Should there be any sort of check for either of these conditions, since using use_accelerate_endpoint incurs higher transfer costs?

That sounds good, but I'm not sure how to implement this.
I can easily know the location of the S3 bucket, but it's harder to get the location of the client. Maybe using its IP address ? But that would work if Girder is reverse proxied.

@pchoisel it might help to explain the problem you're trying to solve, then we may be able to provide better input on design.

@zachmullen Thanks.
A customer wants to enable S3 transfer acceleration for his users that are not in the US to speed up their uploads/downloads to a Girder-based application using an S3 assetstore.
You can enable that by changing the domain name used to connect to S3, but Boto does that automatically if you set use_accelerate_endpoint in the config.

To make this more optimized, I could perhaps use this accelerated endpoint only when Girder sends a transfer link to a user so their client can transfer data to S3, and not when Girder directly transfers data to S3 ?

To make this more optimized, I could perhaps use this accelerated endpoint only when Girder sends a transfer link to a user so their client can transfer data to S3, and not when Girder directly transfers data to S3 ?

I do think this is the right place to make the choice -- at the point of building the presigned URLs rather than bound to the lifetime of the assetstore. I see two possibilities:

1. We attempt to infer the right decision based on properties of the client, e.g. geolocation based on the client IP, and make the choice for them
2. We allow the caller of the REST endpoint(s) to declare that they want to use accelerated transfer, moving the decision point to the client-side or the end users themselves (e.g. a checkbox to enable or disable it under "advanced options" or something).

I think option 2 is a much better idea, but am open to discussion.

I agree that option 2 is the best. However, there are some clients that I cannot change easily #girderwebcomponents
But that's fine, I'll just patch them a bit more. Inferring the location of the client using its IP address seems really shaky.

Do you think I should still add an assetstore option to enable accelerated transfer ? That would make the endpoints return an error if it's disabled but the client requested a accelerated transfer.

Yes, we should probably only allow accelerated transfer if the assetstore explicitly allows it.

Here it is. I originally wanted to store the fact that an upload should be made with acceleration in the upload document, but I was worried about S3 usage being restricted in between the upload init and the chunk upload.

Also, using acceleration just for uploading chunks and using the regular URL for the rest of the requests (completion for example) works fine.

I changed the extraParameters arg of the download API from a param to a jsonParam. I couldn't find anything using it and I think it made more sense. Let me know if I should revert that back.

@manthey If you have some time, could you have a look ?