-
Notifications
You must be signed in to change notification settings - Fork 29
Harvest GitHub-Zenodo DOIs based on latestReleaseDate #6035
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
handle rate limiting
<groupId>io.dockstore</groupId> | ||
<artifactId>swagger-java-zenodo-client</artifactId> | ||
<version>2.1.3</version> | ||
<version>2.1.4</version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This version has created/modified dates in response. See dockstore/swagger-java-zenodo-client#25
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #6035 +/- ##
=============================================
- Coverage 74.45% 74.43% -0.02%
- Complexity 5495 5505 +10
=============================================
Files 381 382 +1
Lines 19786 19839 +53
Branches 2043 2048 +5
=============================================
+ Hits 14731 14767 +36
- Misses 4075 4087 +12
- Partials 980 985 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some minor questions for code comment
dockstore-webservice/src/main/java/io/dockstore/webservice/core/Workflow.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowDAO.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ZenodoHelper.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/helpers/ClientRateLimitHelper.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
|
Description
Use latestReleaseDate to determine what repos to harvest DOIs for. Also added some logic to not run into Zenodo rate limits. The harvesting flow will work like this:
updateDOIs
endpoint.This PR adds a query parameter that specifies how far back to look for GitHub-Zenodo DOIs.
A companion PR for dockstore-deploy will be created to do step 3 daily.
On the rate-limiting, I could not find a library that implements client-side rate limiting based on the headers. I played for a bit with https://github.com/bucket4j/bucket4j, but the Zenodo refresh and/or rate limits don't seem to match be exactly as documented, so when I set up a token bucket, I would end up running into rate limits from Zenodo.
Review Instructions
Issue
SEAB-6688
Security and Privacy
If there are any concerns that require extra attention from the security team, highlight them here and check the box when complete.
e.g. Does this change...
Please make sure that you've checked the following before submitting your pull request. Thanks!
mvn clean install
@RolesAllowed
annotation