ensure that when crawl workflow is updated for a running crawl, the f… #2355
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
I think this might be better suited for the 1.14 release since it's a fairly major change in app behavior. If anything, could separate out just the exclusions work for 1.13.4, but depending on the timing of 1.14 I'm not sure that's worth it. |
3 tasks
…ull config changes *are* propagated, including: - ensure configmap is updated for running crawl - ensure crawler picks up new configmap (remove subPath which prevented dynamic updates) - exclusions changes when adding/removing exclusions - other raw config changes are applied - crawler channel, proxy, timeout, and scale settings are all applied to running crawl - could be first step to pause/resume crawls, but does not require pausing to make changes
c2e81c7 to
24645cd
Compare
ikreymer
added a commit
that referenced
this pull request
Feb 18, 2025
- ensure newly added / remove exclusions are updated in the configmap that is mounted into the crawler volume - exclusions are updated dynamically if crawler pod is running, but when crawler pod is restarted, this ensures new exclusions are also picked up - adds a lastConfigUpdate timestamp to CrawlJob: if lastConfigUpdate in spec is different from current, the configmap is recreated - only updates for exclusion add/remove so far, can later be expanded to other crawler settings (see: #2355 for broader running crawl config updates) - fixes #2408
Member
Author
ikreymer
added a commit
that referenced
this pull request
Feb 19, 2025
…emoved (#2409) exclusions are already updated dynamically if crawler pod is running, but when crawler pod is restarted, this ensures new exclusions are also picked up: - mount configmap in separate path, avoiding subPath, to allow dynamic updates of mounted volume - adds a lastConfigUpdate timestamp to CrawlJob - if lastConfigUpdate in spec is different from current, the configmap is recreated by operator - operator: also update image from channel avoid any issues with updating crawler in channel - only updates for exclusion add/remove so far, can later be expanded to other crawler settings (see: #2355 for broader running crawl config updates) - fixes #2408
emma-sg
pushed a commit
that referenced
this pull request
Feb 19, 2025
…emoved (#2409) exclusions are already updated dynamically if crawler pod is running, but when crawler pod is restarted, this ensures new exclusions are also picked up: - mount configmap in separate path, avoiding subPath, to allow dynamic updates of mounted volume - adds a lastConfigUpdate timestamp to CrawlJob - if lastConfigUpdate in spec is different from current, the configmap is recreated by operator - operator: also update image from channel avoid any issues with updating crawler in channel - only updates for exclusion add/remove so far, can later be expanded to other crawler settings (see: #2355 for broader running crawl config updates) - fixes #2408
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ull config changes are propagated, including: