-
Notifications
You must be signed in to change notification settings - Fork 130
Add KafkaSource migration logic as a post-install job #1889
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add KafkaSource migration logic as a post-install job #1889
Conversation
c5d25b6
to
78d67bc
Compare
Codecov Report
@@ Coverage Diff @@
## main #1889 +/- ##
============================================
+ Coverage 67.08% 67.24% +0.15%
- Complexity 618 632 +14
============================================
Files 135 138 +3
Lines 5757 5837 +80
Branches 178 179 +1
============================================
+ Hits 3862 3925 +63
- Misses 1581 1596 +15
- Partials 314 316 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
The existing KafkaSource creates a deployment for each instance and nothing more. This is what it's in the namespace where I created a `KafkaSouce`. Only the `deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16` is created automatically by the `kafka-controller-manager`, other resources are sink services that shouldn't be deleted during the migration. ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p 1/1 Running 0 106m pod/label-demo 1/1 Running 0 106m pod/label-demo-2 1/1 Running 0 106m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 106m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 106m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 116m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16 1/1 1 1 106m NAME DESIRED CURRENT READY AGE replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8 1 1 1 106m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 106m ``` The plan for post-install logic is to have a generic post-install job in eventing-kafka-broker (in addition to a storage version migrator) this is the job where we place general post-install operations (manifests placed in control-plane/config/post-install). In addition I moved the manifests for the control plane to `control-plane/config/eventing-kafka-broker` to easily produce the `eventing-kafka-broker.yaml` manifest via recursive resolution. *Testing* - Apply `source.yaml` from `eventing-kafka`. - Create `KafkaSouce`(s) - Scale down the `kafka-controller-manager` deployment with: `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0` - Create the post-install manifest with: `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml` - Run the post-install job with: `ko create -f eventing-kafka-post-install.yaml` Output: ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/label-demo 1/1 Running 0 124m pod/label-demo-2 1/1 Running 0 124m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 124m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 124m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 134m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 124m ``` Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
78d67bc
to
6dadacd
Compare
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/approve
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: matzew, pierDipi The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest-required |
/cherry-pick release-1.2 |
/cherry-pick release-1.1 |
/cherry-pick release-1.0 |
@pierDipi: #1889 failed to apply on top of branch "release-1.2":
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@pierDipi: #1889 failed to apply on top of branch "release-1.1":
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@pierDipi: #1889 failed to apply on top of branch "release-1.0":
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
…ions#1889) * Add KafkaSource migration logic as post-install job The existing KafkaSource creates a deployment for each instance and nothing more. This is what it's in the namespace where I created a `KafkaSouce`. Only the `deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16` is created automatically by the `kafka-controller-manager`, other resources are sink services that shouldn't be deleted during the migration. ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p 1/1 Running 0 106m pod/label-demo 1/1 Running 0 106m pod/label-demo-2 1/1 Running 0 106m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 106m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 106m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 116m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16 1/1 1 1 106m NAME DESIRED CURRENT READY AGE replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8 1 1 1 106m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 106m ``` The plan for post-install logic is to have a generic post-install job in eventing-kafka-broker (in addition to a storage version migrator) this is the job where we place general post-install operations (manifests placed in control-plane/config/post-install). In addition I moved the manifests for the control plane to `control-plane/config/eventing-kafka-broker` to easily produce the `eventing-kafka-broker.yaml` manifest via recursive resolution. *Testing* - Apply `source.yaml` from `eventing-kafka`. - Create `KafkaSouce`(s) - Scale down the `kafka-controller-manager` deployment with: `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0` - Create the post-install manifest with: `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml` - Run the post-install job with: `ko create -f eventing-kafka-post-install.yaml` Output: ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/label-demo 1/1 Running 0 124m pod/label-demo-2 1/1 Running 0 124m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 124m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 124m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 134m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 124m ``` Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * use apply, remove generate name Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix channel and source symlinks Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix style check Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
…ions#1889) * Add KafkaSource migration logic as post-install job The existing KafkaSource creates a deployment for each instance and nothing more. This is what it's in the namespace where I created a `KafkaSouce`. Only the `deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16` is created automatically by the `kafka-controller-manager`, other resources are sink services that shouldn't be deleted during the migration. ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p 1/1 Running 0 106m pod/label-demo 1/1 Running 0 106m pod/label-demo-2 1/1 Running 0 106m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 106m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 106m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 116m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16 1/1 1 1 106m NAME DESIRED CURRENT READY AGE replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8 1 1 1 106m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 106m ``` The plan for post-install logic is to have a generic post-install job in eventing-kafka-broker (in addition to a storage version migrator) this is the job where we place general post-install operations (manifests placed in control-plane/config/post-install). In addition I moved the manifests for the control plane to `control-plane/config/eventing-kafka-broker` to easily produce the `eventing-kafka-broker.yaml` manifest via recursive resolution. *Testing* - Apply `source.yaml` from `eventing-kafka`. - Create `KafkaSouce`(s) - Scale down the `kafka-controller-manager` deployment with: `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0` - Create the post-install manifest with: `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml` - Run the post-install job with: `ko create -f eventing-kafka-post-install.yaml` Output: ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/label-demo 1/1 Running 0 124m pod/label-demo-2 1/1 Running 0 124m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 124m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 124m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 134m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 124m ``` Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * use apply, remove generate name Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix channel and source symlinks Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix style check Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
* Add KafkaSource migration logic as a post-install job (#1889) * Add KafkaSource migration logic as post-install job The existing KafkaSource creates a deployment for each instance and nothing more. This is what it's in the namespace where I created a `KafkaSouce`. Only the `deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16` is created automatically by the `kafka-controller-manager`, other resources are sink services that shouldn't be deleted during the migration. ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p 1/1 Running 0 106m pod/label-demo 1/1 Running 0 106m pod/label-demo-2 1/1 Running 0 106m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 106m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 106m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 116m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16 1/1 1 1 106m NAME DESIRED CURRENT READY AGE replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8 1 1 1 106m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 106m ``` The plan for post-install logic is to have a generic post-install job in eventing-kafka-broker (in addition to a storage version migrator) this is the job where we place general post-install operations (manifests placed in control-plane/config/post-install). In addition I moved the manifests for the control plane to `control-plane/config/eventing-kafka-broker` to easily produce the `eventing-kafka-broker.yaml` manifest via recursive resolution. *Testing* - Apply `source.yaml` from `eventing-kafka`. - Create `KafkaSouce`(s) - Scale down the `kafka-controller-manager` deployment with: `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0` - Create the post-install manifest with: `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml` - Run the post-install job with: `ko create -f eventing-kafka-post-install.yaml` Output: ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/label-demo 1/1 Running 0 124m pod/label-demo-2 1/1 Running 0 124m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 124m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 124m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 134m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 124m ``` Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * use apply, remove generate name Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix channel and source symlinks Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix style check Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Add Post-install to released artifacts (#1902) Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Delete global resources during KafkaSource migration (#1896) * Add KafkaSource migration logic as post-install job The existing KafkaSource creates a deployment for each instance and nothing more. This is what it's in the namespace where I created a `KafkaSouce`. Only the `deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16` is created automatically by the `kafka-controller-manager`, other resources are sink services that shouldn't be deleted during the migration. ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p 1/1 Running 0 106m pod/label-demo 1/1 Running 0 106m pod/label-demo-2 1/1 Running 0 106m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 106m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 106m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 116m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16 1/1 1 1 106m NAME DESIRED CURRENT READY AGE replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8 1 1 1 106m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 106m ``` The plan for post-install logic is to have a generic post-install job in eventing-kafka-broker (in addition to a storage version migrator) this is the job where we place general post-install operations (manifests placed in control-plane/config/post-install). In addition I moved the manifests for the control plane to `control-plane/config/eventing-kafka-broker` to easily produce the `eventing-kafka-broker.yaml` manifest via recursive resolution. *Testing* - Apply `source.yaml` from `eventing-kafka`. - Create `KafkaSouce`(s) - Scale down the `kafka-controller-manager` deployment with: `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0` - Create the post-install manifest with: `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml` - Run the post-install job with: `ko create -f eventing-kafka-post-install.yaml` Output: ``` pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all NAME READY STATUS RESTARTS AGE pod/label-demo 1/1 Running 0 124m pod/label-demo-2 1/1 Running 0 124m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/event-display ClusterIP 10.96.16.19 <none> 80/TCP 124m service/event-display-2 ClusterIP 10.96.177.33 <none> 80/TCP 124m service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 134m NAME TOPICS BOOTSTRAPSERVERS READY REASON AGE kafkasource.sources.knative.dev/kafka-source ["knative-demo-topic"] ["my-cluster-kafka-bootstrap.kafka:9092"] True 124m ``` Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * use apply, remove generate name Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix channel and source symlinks Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Fix style check Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com> * Delete global resources during KafkaSource migration - Delete mutatingwebhookconfigurations.admissionregistration.k8s.io/defaulting.webhook.kafka.sources.knative.dev - Delete validatingwebhookconfigurations.admissionregistration.k8s.io/validation.webhook.kafka.sources.knative.dev - Delete validatingwebhookconfigurations.admissionregistration.k8s.io/config.webhook.kafka.sources.knative.dev - Delete service knative-eventing/kafka-source-webhook - Delete Service knative-eventing/kafka-controller - Delete deployment.apps/kafka-controller-manager - Delete ClusterRoleBinding/eventing-sources-kafka-controller - Delete ClusterRoleBinding/eventing-sources-kafka-controller-addressable-resolver - Delete ClusterRole/eventing-sources-kafka-controller - Delete ServiceAccount knative-eventing/kafka-controller-manager - Delete ConfigMap knative-eventing/config-kafka-source-defaults - Delete Secret knative-eventing/kafka-source-webhook-certs Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
The existing KafkaSource creates a deployment for each instance
and nothing more.
This is what it's in the namespace where I created a
KafkaSouce
.Only the
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16
is created automatically by the
kafka-controller-manager
, other resources are sink servicesthat shouldn't be deleted during the migration.
The plan for post-install logic is to have a generic post-install
job in eventing-kafka-broker (in addition to a storage version
migrator) this is the job where we place general post-install
operations (manifests placed in control-plane/config/post-install).
In addition I moved the manifests for the control plane to
control-plane/config/eventing-kafka-broker
to easily producethe
eventing-kafka-broker.yaml
manifest via recursiveresolution.
Testing
source.yaml
fromeventing-kafka
.KafkaSouce
(s)kafka-controller-manager
deployment with:k scale deployment -n knative-eventing kafka-controller-manager --replicas=0
ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml
ko create -f eventing-kafka-post-install.yaml
Output:
Signed-off-by: Pierangelo Di Pilato pierdipi@redhat.com
Proposed Changes
Release Note
Next: left-over global resources deletion