Chained notifications have empty body #2304
Description
(this issue might be related to #2226 and/or #1583 (comment))
I have alerts with chained notifications that look like this (you can probably completely ignore the mail notifications here, they work fine. It's the HTTP POST notifications that don't work correctly):
template notification {
body1 = BODY1
body2 = BODY2
}
notification mail2 {
print = true
runOnActions = false
email = ...
timeout = 1m
next = null
}
notification mail1 {
print = true
runOnActions = false
email = ...
timeout = 1m
next = mail2
}
notification phone2 {
print = true
runOnActions = false
post = https://...
bodyTemplate = body2
timeout = 1m
next = null
}
notification phone1 {
print = true
runOnActions = false
post = https://...
bodyTemplate = body1
timeout = 1m
next = phone2
}
lookup notifications {
entry environment=staging {
crit = null
warn = null
}
entry environment=production {
crit = mail1,phone1
warn = mail1
}
entry environment=* {
crit = null
warn = null
}
}
template alert1 {
inherit = notification
subject = ...
body = ...
}
alert alert1 {
template = alert1
warn = ...
crit = ...
critNotification = lookup("notifications", "crit")
warnNotification = lookup("notifications", "warn")
}
I have a low timeout (1m) for debug purposes. Longer timeouts behave the same.
When a critical alert is triggered, the first notifications (phone1 and mail1 works as expected. However, if don't acknowledge the alert, and let go on to the next notification (phone2), the HTTP POST request is performed with an empty body (size 0).
Look at the logs below:
UTC 180910 13:43:25.777info: notify.go:59: alert1/critical
UTC 180910 13:43:25.777info: notify.go:251: relayed email alert1{...} to [...] sucessfully. Subject: 44 bytes. Body: 1089 bytes.
UTC 180910 13:43:26.321 info: notify.go:59: Subject: alert1/critical, Body: BODY1
UTC 180910 13:44:25.572 info: notify.go:59: alert1/critical
UTC 180910 13:44:25.572 info: notify.go:251: relayed email alert1{...} to [...] sucessfully. Subject: 44 bytes. Body: 1089 bytes.
UTC 180910 13:44:25.905 info: notify.go:59: Subject: alert1/critical, Body:
UTC 180910 13:44:25.905 error: notify.go:54: sending http: bad response for 'phone2' alert notification using template key 'body2' for alert keys alert1{...} method POST: 400
Look at the two last lines. Bosun correctly picked the body2 field, but it never set it to BODY2 as expected, instead it just POSTed an empty body, causing a 400 bad request for the HTTP endpoint.