feat: add better handler for part size #214
Conversation
|
@m90 Hello, I think it works. Let me know if you don't have any side effects ! |
m90
left a comment
There was a problem hiding this comment.
Thanks for working on this. I left a few comments, if you have any questions please inquire further. Also, CI is currently not passing when storing data in MinIO, so there seems to be something in the code that breaks the existing behavior. Let me know if you need help debugging this.
README.md
Outdated
| - [Using mysqldump to prepare the backup](#using-mysqldump-to-prepare-the-backup) | ||
| - [Running multiple instances in the same setup](#running-multiple-instances-in-the-same-setup) | ||
| - [Differences to `futurice/docker-volume-backup`](#differences-to-futuricedocker-volume-backup) | ||
| - [docker-volume-backup](#docker-volume-backup) |
There was a problem hiding this comment.
Are these changes to the TOC accidental?
There was a problem hiding this comment.
This somehow resurfaced, could you undo the changes again?
|
Hello, I saw your review ! Btw, about the upgrade of Minio. |
In case it's not related to the feature, I'd prefer to stick to the version |
fix: use local file fix: try with another path fix: use bytes chore: go back go back readme goback goback goback
|
Hello ! I got time to continue, Let me know if everything's okay! |
m90
left a comment
There was a problem hiding this comment.
This looks good now, I think I spotted one logic mistake though. Let me know if that makes sense.
internal/storage/s3/s3.go
Outdated
| }); err != nil { | ||
| } | ||
|
|
||
| if b.partSize < 0 { |
There was a problem hiding this comment.
Wouldn't this need to be
if b.partSize > 0 {
README.md
Outdated
| # While Minio uses a hard coded value to 10.000. As a workaround, try to set a higher value. | ||
| # Defaults to "16" (MB) if unset (from minio), you can set this value according to your needs. | ||
|
|
||
| # AWS_PART_SIZE= |
There was a problem hiding this comment.
| # AWS_PART_SIZE= | |
| # AWS_PART_SIZE=1000 |
I think it'd be good to set an example value here so people know how the number is expected to be formatted. E.g. right now I'm not sure if I'm expected to pass 1.000 or 1000.
There was a problem hiding this comment.
Okay, I'll take all in consideration and I will commit as soon as I can !
|
Done ! I'm doing my tests, let me know if yours pass also ! |
|
Worked ! |
|
This is now released as v2.28.0 - thanks for your contribution 🎩 |
Hello,
Here, a potential PR to allow a better handling with part size.
The problem solved is when uploading large files, the part size can be limiting.
In my case, I use S3 from scaleway and the counter size is set to 1.000 where in AWS is 10.000.
It can be problematic since Minio hard coded this. So by setting, at least, a size part I can prevent this value to reach or get over 1000.
So, what I changed :
putparameter to copy the file.I am not a go dev, so let me know if I need to change stuff.
I am testing the PR in multiple cases, so I will let you know when it works for me.
Let me know also, if you try, if it works.
EDIT : It doesn't work as it, so it is stil a draft.