-
Notifications
You must be signed in to change notification settings - Fork 245
Description
Describe the bug
After updating the tridentprovisioner and add silenceAutosupport: true the configured trident-csi-token went missing. It's unclear how this happened. The trident-csi pods kept running, complaining about the missing secret:
2m6s Warning FailedMount pod/trident-csi-74d6n MountVolume.SetUp failed for volume "trident-csi-token-qkdm2" : secret "trident-csi-token-qkdm2" not found
The token was not present any more, but there were 2 other tokens present. 1 unused.
Turned out that trident-csi-token-29c5f was the "correct" one used after killing the pods later.
trident-csi-token-29c5f kubernetes.io/service-account-token 4 2d16h
trident-csi-token-jjsjt kubernetes.io/service-account-token 4 2d16h
See discussion in Netapp slack https://netapppub.slack.com/archives/C1E3QH84C/p1599221156108100
Trident operation was prevented because communication of trident main with the csi pods was not possible any more.
Manual workaround: Deleting all daemonset pods. They were recreated with the correct secret.
Environment
- Trident version: 20.07.0
- Trident installation flags used: CR: debug: true
- Container runtime: cri-o
- Kubernetes version: 1.18
- Kubernetes orchestrator: OpenShift v4.5.7
- Kubernetes enabled feature gates: -
- OS: RHCOS
- NetApp backend types: ontap-nas, ontap-nas-economy, ontap-san
- Other:
To Reproduce
Unsure. Possible causes: Deletion of the secret or deployment. See slack discussion.
Expected behavior
In general: The operator should be able to handle and correct such events.
Suggested in Slack:
Operator should re-create the daemonset as well as the deployment pods if the service account is re-created.
Or in each reconcile loop operator should verify that the secrets in the pods associated with the daemonset and the deployment are correct and matches the service account secrets.