-
-
Notifications
You must be signed in to change notification settings - Fork 4.6k
Implement backend for FileJump (#8114) #8693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
- Implemented complete filejump backend with API integration - Added comprehensive test suite for filejump backend - Updated documentation and configuration - All integration tests passing for filejump backend
How are the integration tests going? Does See here for more info: https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-a-new-backend |
Yes, everything works. I just tested it again. But I had to start it with a bigger timeout. I tested it with It finished in 1h48min. Here is the output of the test: |
That is a bit slow, but not the slowest backend!
That looks really good. The bisync tests took the longest. We can add a time modifier in the rclone/fstest/test_all/config.yaml Line 201 in ff9cbab
This multiplies the normal timeout (which is 1h) by that factor. Looking at the timings none of the tests took longer than 2h so that should be fine. This backend was made using the public REST API at https://eu.filejump.com/api-docs - is that right? I'll give the code a review now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking really great. The integration tests all passing is fantastic.
I've put some notes inline - nothing serious.
I think you've got some lint things to fix too.
Thank you
func (i *Item) ModTime() (t time.Time) { | ||
// Parse UpdatedAt first | ||
if i.UpdatedAt != "" { | ||
// Try multiple time formats that FileJump might use |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need multiple formats?
If so you should probably factor this parsing out as I can see it repeated below
Options: []fs.Option{{ | ||
Name: "access_token", | ||
Help: "You should create an API access token here: https://drive.filejump.com/account-settings", | ||
Required: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs Sensitive: true
hasMetaData bool | ||
size int64 | ||
modTime time.Time | ||
preserveModTime bool // Flag to indicate if modTime should be preserved |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this for?
return err | ||
} | ||
*/ | ||
func CallJSON[T any, B any](f *Fs, ctx context.Context, method string, path string, params *url.Values, body *B) (*T, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting use of generics. I don't think they are strictly needed, but I see it allows you return the *T
from the function (rather than pass it in) which looks nice.
Path: path, | ||
ExtraHeaders: map[string]string{ | ||
"Accept": "application/json", | ||
"Accept-Encoding": "gzip, deflate, br, zstd", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can Go deal with those other formats? I only ever use gzip
.
|
||
### Restricted filenames | ||
|
||
FileJump has a minimum filename length requirement of 3 characters. Rclone will automatically pad shorter filenames with spaces to meet this requirement. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is the first time I've seen that restriction in a cloud storage system!
I think this needs to be a little bit clearer
Rclone will automatically pad shorter filenames with spaces on upload to meet this requirement. It will also remove the spaces on download. This means if you happen to have a 1 or 2 character file name ending in a space then syncing may go wrong.
// FileJump doesn't support setting custom modification times via API | ||
// The modification time is set by the server when the file is uploaded | ||
// Return this error to indicate that setting mod time requires re-upload | ||
return fs.ErrorCantSetModTimeWithoutDelete |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should just return fs.ErrorCantSetModTime
here. The fs.ErrorCantSetModTimeWithoutDelete
is for backends which can only set modification times when they upload a file.
|
||
- FileJump doesn't support setting custom modification times | ||
- Upload time is used as the modification time | ||
- `--update` flag behavior may be affected |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using the --update
flag can improve sync accuracy.
iDir, _ := strconv.Atoi(rootID) | ||
request := RequestDelete{ | ||
EntryIds: []int{iDir}, | ||
DeleteForever: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally backends default to storing deletes in the trash and use a hard_delete
flag to override.
Eg
--jottacloud-hard-delete Delete files permanently rather than putting them into the trash
--mega-hard-delete Delete files permanently rather than putting them into the trash
We started doing it that way round after one user deleted all of their files by accident! So safety first, but overridable by the user.
@@ -162,6 +162,8 @@ backends: | |||
- backend: "gofile" | |||
remote: "TestGoFile:" | |||
fastlist: true | |||
- backend: "filejump" | |||
remote: "TestFileJump:" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might want extra time here as discussed above!
Yes, that is right. But Google Chrome even shows the typescript of the original Code of the website and it is easy to see in the network tab in the browser after every request, what exactly happened. So there are two different ways how to upload a file. One is by requesting an upload url and upload it there (directly to the underlying wasabi backend). The website does it this way. And the other way to upload a file is via the public REST API. It looks like a wrapper, which uploads the file later to the wasabi backend. I did it like the website. With big files, it maybe should be faster, because the files don't use the wrapper. But maybe for short files, I should use the REST API. As I am at the moment on holidays and can just program on my phone, I will look into your change review later. But actually this artificial intelligence agent: https://www.atlassian.com/blog/announcements/rovo-dev-command-line-interface helped me A LOT with programming. I told it to make my code pass the tests and the same day, my code passed ALL the tests! Sometimes, I had to guide it, but it is unbelievable, you should test it 😄 It is completely free until the end of this month. And if you like, while I am on holidays, you can update my code, however you like. But don't worry, just if you have time |
@masrlinu thanks for the update. I'll mark this for v1.71 which is due 23rd August. So enjoy your holiday and don't feel you need to look at this until you get back :-) |
What is the purpose of this change?
This PR implements a backend for the storage provider FileJump.com.
Was the change discussed in an issue or in the forum before?
Yes, this backend was requested in issue #8114.
Checklist