This repository was archived by the owner on Feb 13, 2025. It is now read-only.
This repository was archived by the owner on Feb 13, 2025. It is now read-only.
Bug: Duplication of chained notifications after api call for reload config #2444
Description
Expected behavior
API POST /api/reload -d '{ "Reload": true }' should reread config and make full restart of scheduler
Current behaviour
When we run bosun first time we have 1 dispatchNotifications routine:
goroutine 72 [select]:
bosun.org/cmd/bosun/sched.(*Schedule).dispatchNotifications(0x22d9500)
bosun.org/cmd/bosun/sched/notify.go:26 +0x140
created by bosun.org/cmd/bosun/sched.(*Schedule).Run
bosun.org/cmd/bosun/sched/alertRunner.go:18 +0xa4
After reload through api POST /api/reload -d '{ "Reload": true }':
goroutine 72 [select]:
bosun.org/cmd/bosun/sched.(*Schedule).dispatchNotifications(0x22d9500)
bosun.org/cmd/bosun/sched/notify.go:26 +0x140
created by bosun.org/cmd/bosun/sched.(*Schedule).Run
bosun.org/cmd/bosun/sched/alertRunner.go:18 +0xa4
...
goroutine 177 [select]:
bosun.org/cmd/bosun/sched.(*Schedule).dispatchNotifications(0xc00032c000)
bosun.org/cmd/bosun/sched/notify.go:26 +0x140
created by bosun.org/cmd/bosun/sched.(*Schedule).Run
bosun.org/cmd/bosun/sched/alertRunner.go:18 +0xa4
As result, we have duplicate chained notifications
Steps to reproduce
Please provide detailed steps for reproducing the issue.
- Run
bosundaemon with system config:
...
HTTPListen = ":8071"
EnableReload = true
...
And rules definition:
alert test {
template = default_template
$flapper = ((epoch() % 120 ) >= 60)
crit = $flapper
warn = !$flapper
runEvery = 1
critNotification = default
warnNotification = default
}
template default_template {
subject = Test
body = ''
postBody = `{
"body": "{{.Subject}}"
}`
}
notification default_notification {
post = http://127.0.0.1:8000/
bodyTemplate = body
next = default_notification
timeout = 1m
}
- Run notification service:
#!/usr/bin/env python3
from http.server import HTTPServer, BaseHTTPRequestHandler
class S(BaseHTTPRequestHandler):
def _set_headers(self):
self.send_response(200)
self.end_headers()
def do_POST(self):
# Doesn't do anything with posted data
self._set_headers()
self.wfile.write("ok")
def run(server_class=HTTPServer, handler_class=S, addr="localhost", port=8000):
server_address = (addr, port)
httpd = server_class(server_address, handler_class)
print(f"Starting httpd server on {addr}:{port}")
httpd.serve_forever()
if __name__ == "__main__":
run()
- Wait for first notification. After that reload config -
curl -XPOST 127.0.0.1:8071/api/reload -d '{ "Reload": true }' - Check duplicate call in notification service output
Context
Running commit: 81e993e
OS: linux_x64, Linux 5.3.14-300.fc31.x86_64
Logs
Nothing interesting in logs...