这是indexloc提供的服务,不要输入任何密码
Skip to content

Missing secret after operator configuration update #444

@megabreit

Description

@megabreit

Describe the bug
After updating the tridentprovisioner and add silenceAutosupport: true the configured trident-csi-token went missing. It's unclear how this happened. The trident-csi pods kept running, complaining about the missing secret:

2m6s        Warning   FailedMount   pod/trident-csi-74d6n        MountVolume.SetUp failed for volume "trident-csi-token-qkdm2" : secret "trident-csi-token-qkdm2" not found

The token was not present any more, but there were 2 other tokens present. 1 unused.
Turned out that trident-csi-token-29c5f was the "correct" one used after killing the pods later.

trident-csi-token-29c5f                    kubernetes.io/service-account-token   4      2d16h
trident-csi-token-jjsjt                    kubernetes.io/service-account-token   4      2d16h

See discussion in Netapp slack https://netapppub.slack.com/archives/C1E3QH84C/p1599221156108100

Trident operation was prevented because communication of trident main with the csi pods was not possible any more.

Manual workaround: Deleting all daemonset pods. They were recreated with the correct secret.

Environment

  • Trident version: 20.07.0
  • Trident installation flags used: CR: debug: true
  • Container runtime: cri-o
  • Kubernetes version: 1.18
  • Kubernetes orchestrator: OpenShift v4.5.7
  • Kubernetes enabled feature gates: -
  • OS: RHCOS
  • NetApp backend types: ontap-nas, ontap-nas-economy, ontap-san
  • Other:

To Reproduce
Unsure. Possible causes: Deletion of the secret or deployment. See slack discussion.

Expected behavior
In general: The operator should be able to handle and correct such events.
Suggested in Slack:
Operator should re-create the daemonset as well as the deployment pods if the service account is re-created.
Or in each reconcile loop operator should verify that the secrets in the pods associated with the daemonset and the deployment are correct and matches the service account secrets.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions