这是indexloc提供的服务,不要输入任何密码
Skip to content

Add KafkaSource migration logic as a post-install job #1889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

pierDipi
Copy link
Member

@pierDipi pierDipi commented Feb 10, 2022

The existing KafkaSource creates a deployment for each instance
and nothing more.

This is what it's in the namespace where I created a KafkaSouce.
Only the
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16
is created automatically by the
kafka-controller-manager, other resources are sink services
that shouldn't be deleted during the migration.

pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p   1/1     Running   0          106m
pod/label-demo                                                        1/1     Running   0          106m
pod/label-demo-2                                                      1/1     Running   0          106m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    106m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    106m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   116m

NAME                                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16   1/1     1            1           106m

NAME                                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8   1         1         1       106m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             106m

The plan for post-install logic is to have a generic post-install
job in eventing-kafka-broker (in addition to a storage version
migrator) this is the job where we place general post-install
operations (manifests placed in control-plane/config/post-install).

In addition I moved the manifests for the control plane to
control-plane/config/eventing-kafka-broker to easily produce
the eventing-kafka-broker.yaml manifest via recursive
resolution.

Testing

  • Apply source.yaml from eventing-kafka.
  • Create KafkaSouce(s)
  • Scale down the kafka-controller-manager deployment with:
    k scale deployment -n knative-eventing kafka-controller-manager --replicas=0
  • Create the post-install manifest with:
    ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml
  • Run the post-install job with:
    ko create -f eventing-kafka-post-install.yaml

Output:

pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME               READY   STATUS    RESTARTS   AGE
pod/label-demo     1/1     Running   0          124m
pod/label-demo-2   1/1     Running   0          124m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    124m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    124m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   134m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             124m

Signed-off-by: Pierangelo Di Pilato pierdipi@redhat.com

Proposed Changes

  • Add KafkaSource migration logic as a post-install job

Release Note

Add KafkaSource migration logic as a post-install job (`eventing-kafka-post-install.yaml`).

Next: left-over global resources deletion

@knative-prow-robot knative-prow-robot added area/control-plane size/L Denotes a PR that changes 100-499 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 10, 2022
@pierDipi pierDipi force-pushed the kafka-source-migration-post-install branch from c5d25b6 to 78d67bc Compare February 10, 2022 11:26
@codecov
Copy link

codecov bot commented Feb 10, 2022

Codecov Report

Merging #1889 (f4c40b9) into main (e685853) will increase coverage by 0.15%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##               main    #1889      +/-   ##
============================================
+ Coverage     67.08%   67.24%   +0.15%     
- Complexity      618      632      +14     
============================================
  Files           135      138       +3     
  Lines          5757     5837      +80     
  Branches        178      179       +1     
============================================
+ Hits           3862     3925      +63     
- Misses         1581     1596      +15     
- Partials        314      316       +2     
Flag Coverage Δ
java-unittests 82.22% <ø> (-0.14%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...e/eventing/kafka/broker/core/file/FileWatcher.java 75.64% <0.00%> (-5.13%) ⬇️
...ispatcher/impl/http/WebClientCloudEventSender.java 53.44% <0.00%> (-0.94%) ⬇️
...lane/pkg/reconciler/consumergroup/consumergroup.go 75.92% <0.00%> (ø)
.../kafka/broker/dispatcher/impl/ResourceContext.java 81.81% <0.00%> (ø)
...oker/dispatcher/impl/ResponseFailureException.java 0.00% <0.00%> (ø)
.../broker/dispatcher/impl/ConsumerRecordContext.java 100.00% <0.00%> (ø)
...r/dispatcher/main/ConsumerVerticleFactoryImpl.java 87.93% <0.00%> (+0.10%) ⬆️
...ve/eventing/kafka/broker/core/metrics/Metrics.java 87.30% <0.00%> (+1.58%) ⬆️
...a/broker/dispatcher/impl/RecordDispatcherImpl.java 83.21% <0.00%> (+4.48%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e685853...f4c40b9. Read the comment docs.

The existing KafkaSource creates a deployment for each instance
and nothing more.

This is what it's in the namespace where I created a `KafkaSouce`.
Only the
`deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16`
is created automatically by the
`kafka-controller-manager`, other resources are sink services
that shouldn't be deleted during the migration.
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p   1/1     Running   0          106m
pod/label-demo                                                        1/1     Running   0          106m
pod/label-demo-2                                                      1/1     Running   0          106m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    106m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    106m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   116m

NAME                                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16   1/1     1            1           106m

NAME                                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8   1         1         1       106m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             106m
```

The plan for post-install logic is to have a generic post-install
job in eventing-kafka-broker (in addition to a storage version
migrator) this is the job where we place general post-install
operations (manifests placed in control-plane/config/post-install).

In addition I moved the manifests for the control plane to
`control-plane/config/eventing-kafka-broker` to easily produce
the `eventing-kafka-broker.yaml` manifest via recursive
resolution.

*Testing*

- Apply `source.yaml` from `eventing-kafka`.
- Create `KafkaSouce`(s)
- Scale down the `kafka-controller-manager` deployment with:
  `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0`
- Create the post-install manifest with:
  `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml`
- Run the post-install job with:
  `ko create -f eventing-kafka-post-install.yaml`

Output:
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME               READY   STATUS    RESTARTS   AGE
pod/label-demo     1/1     Running   0          124m
pod/label-demo-2   1/1     Running   0          124m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    124m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    124m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   134m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             124m
```

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
@pierDipi pierDipi force-pushed the kafka-source-migration-post-install branch from 78d67bc to 6dadacd Compare February 10, 2022 11:32
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
Copy link
Contributor

@matzew matzew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@knative-prow-robot knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 14, 2022
@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pierDipi
Copy link
Member Author

/retest-required

@knative-prow-robot knative-prow-robot merged commit 345d57c into knative-extensions:main Feb 14, 2022
@pierDipi pierDipi deleted the kafka-source-migration-post-install branch February 14, 2022 15:38
@pierDipi
Copy link
Member Author

/cherry-pick release-1.2

@pierDipi
Copy link
Member Author

/cherry-pick release-1.1

@pierDipi
Copy link
Member Author

/cherry-pick release-1.0

@knative-prow-robot
Copy link
Contributor

@pierDipi: #1889 failed to apply on top of branch "release-1.2":

Applying: Add KafkaSource migration logic as post-install job
error: mode change for control-plane/config/200-controller/100-config-kafka-descheduler.yaml, which is not in current HEAD
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Add KafkaSource migration logic as post-install job
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@knative-prow-robot
Copy link
Contributor

@pierDipi: #1889 failed to apply on top of branch "release-1.1":

Applying: Add KafkaSource migration logic as post-install job
error: mode change for control-plane/config/100-kafka-internal/100-consumer.yaml, which is not in current HEAD
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Add KafkaSource migration logic as post-install job
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@knative-prow-robot
Copy link
Contributor

@pierDipi: #1889 failed to apply on top of branch "release-1.0":

Applying: Add KafkaSource migration logic as post-install job
error: mode change for control-plane/config/100-channel/100-kafka-channel-configmap.yaml, which is not in current HEAD
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Add KafkaSource migration logic as post-install job
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pierDipi added a commit to pierDipi/eventing-kafka-broker that referenced this pull request Feb 15, 2022
…ions#1889)

* Add KafkaSource migration logic as post-install job

The existing KafkaSource creates a deployment for each instance
and nothing more.

This is what it's in the namespace where I created a `KafkaSouce`.
Only the
`deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16`
is created automatically by the
`kafka-controller-manager`, other resources are sink services
that shouldn't be deleted during the migration.
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p   1/1     Running   0          106m
pod/label-demo                                                        1/1     Running   0          106m
pod/label-demo-2                                                      1/1     Running   0          106m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    106m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    106m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   116m

NAME                                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16   1/1     1            1           106m

NAME                                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8   1         1         1       106m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             106m
```

The plan for post-install logic is to have a generic post-install
job in eventing-kafka-broker (in addition to a storage version
migrator) this is the job where we place general post-install
operations (manifests placed in control-plane/config/post-install).

In addition I moved the manifests for the control plane to
`control-plane/config/eventing-kafka-broker` to easily produce
the `eventing-kafka-broker.yaml` manifest via recursive
resolution.

*Testing*

- Apply `source.yaml` from `eventing-kafka`.
- Create `KafkaSouce`(s)
- Scale down the `kafka-controller-manager` deployment with:
  `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0`
- Create the post-install manifest with:
  `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml`
- Run the post-install job with:
  `ko create -f eventing-kafka-post-install.yaml`

Output:
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME               READY   STATUS    RESTARTS   AGE
pod/label-demo     1/1     Running   0          124m
pod/label-demo-2   1/1     Running   0          124m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    124m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    124m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   134m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             124m
```

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* use apply, remove generate name

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix channel and source symlinks

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix style check

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
pierDipi added a commit to pierDipi/eventing-kafka-broker that referenced this pull request Feb 16, 2022
…ions#1889)

* Add KafkaSource migration logic as post-install job

The existing KafkaSource creates a deployment for each instance
and nothing more.

This is what it's in the namespace where I created a `KafkaSouce`.
Only the
`deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16`
is created automatically by the
`kafka-controller-manager`, other resources are sink services
that shouldn't be deleted during the migration.
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p   1/1     Running   0          106m
pod/label-demo                                                        1/1     Running   0          106m
pod/label-demo-2                                                      1/1     Running   0          106m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    106m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    106m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   116m

NAME                                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16   1/1     1            1           106m

NAME                                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8   1         1         1       106m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             106m
```

The plan for post-install logic is to have a generic post-install
job in eventing-kafka-broker (in addition to a storage version
migrator) this is the job where we place general post-install
operations (manifests placed in control-plane/config/post-install).

In addition I moved the manifests for the control plane to
`control-plane/config/eventing-kafka-broker` to easily produce
the `eventing-kafka-broker.yaml` manifest via recursive
resolution.

*Testing*

- Apply `source.yaml` from `eventing-kafka`.
- Create `KafkaSouce`(s)
- Scale down the `kafka-controller-manager` deployment with:
  `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0`
- Create the post-install manifest with:
  `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml`
- Run the post-install job with:
  `ko create -f eventing-kafka-post-install.yaml`

Output:
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME               READY   STATUS    RESTARTS   AGE
pod/label-demo     1/1     Running   0          124m
pod/label-demo-2   1/1     Running   0          124m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    124m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    124m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   134m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             124m
```

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* use apply, remove generate name

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix channel and source symlinks

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix style check

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
knative-prow-robot pushed a commit that referenced this pull request Feb 16, 2022
* Add KafkaSource migration logic as a post-install job (#1889)

* Add KafkaSource migration logic as post-install job

The existing KafkaSource creates a deployment for each instance
and nothing more.

This is what it's in the namespace where I created a `KafkaSouce`.
Only the
`deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16`
is created automatically by the
`kafka-controller-manager`, other resources are sink services
that shouldn't be deleted during the migration.
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p   1/1     Running   0          106m
pod/label-demo                                                        1/1     Running   0          106m
pod/label-demo-2                                                      1/1     Running   0          106m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    106m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    106m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   116m

NAME                                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16   1/1     1            1           106m

NAME                                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8   1         1         1       106m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             106m
```

The plan for post-install logic is to have a generic post-install
job in eventing-kafka-broker (in addition to a storage version
migrator) this is the job where we place general post-install
operations (manifests placed in control-plane/config/post-install).

In addition I moved the manifests for the control plane to
`control-plane/config/eventing-kafka-broker` to easily produce
the `eventing-kafka-broker.yaml` manifest via recursive
resolution.

*Testing*

- Apply `source.yaml` from `eventing-kafka`.
- Create `KafkaSouce`(s)
- Scale down the `kafka-controller-manager` deployment with:
  `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0`
- Create the post-install manifest with:
  `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml`
- Run the post-install job with:
  `ko create -f eventing-kafka-post-install.yaml`

Output:
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME               READY   STATUS    RESTARTS   AGE
pod/label-demo     1/1     Running   0          124m
pod/label-demo-2   1/1     Running   0          124m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    124m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    124m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   134m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             124m
```

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* use apply, remove generate name

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix channel and source symlinks

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix style check

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Add Post-install to released artifacts (#1902)

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Delete global resources during KafkaSource migration (#1896)

* Add KafkaSource migration logic as post-install job

The existing KafkaSource creates a deployment for each instance
and nothing more.

This is what it's in the namespace where I created a `KafkaSouce`.
Only the
`deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16`
is created automatically by the
`kafka-controller-manager`, other resources are sink services
that shouldn't be deleted during the migration.
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8bhp5p   1/1     Running   0          106m
pod/label-demo                                                        1/1     Running   0          106m
pod/label-demo-2                                                      1/1     Running   0          106m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    106m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    106m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   116m

NAME                                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16   1/1     1            1           106m

NAME                                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kafkasource-kafka-source-2e0b5e79-8547-46e8-b018-aac56c6c8d16-695fbb68b8   1         1         1       106m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             106m
```

The plan for post-install logic is to have a generic post-install
job in eventing-kafka-broker (in addition to a storage version
migrator) this is the job where we place general post-install
operations (manifests placed in control-plane/config/post-install).

In addition I moved the manifests for the control plane to
`control-plane/config/eventing-kafka-broker` to easily produce
the `eventing-kafka-broker.yaml` manifest via recursive
resolution.

*Testing*

- Apply `source.yaml` from `eventing-kafka`.
- Create `KafkaSouce`(s)
- Scale down the `kafka-controller-manager` deployment with:
  `k scale deployment -n knative-eventing kafka-controller-manager --replicas=0`
- Create the post-install manifest with:
  `ko resolve -Rf control-plane/config/post-install/ > eventing-kafka-post-install.yaml`
- Run the post-install job with:
  `ko create -f eventing-kafka-post-install.yaml`

Output:
```
pierdipi@pierdipi eventing-kafka-broker (kafka-source-migration-post-install) $ k get all
NAME               READY   STATUS    RESTARTS   AGE
pod/label-demo     1/1     Running   0          124m
pod/label-demo-2   1/1     Running   0          124m

NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/event-display     ClusterIP   10.96.16.19    <none>        80/TCP    124m
service/event-display-2   ClusterIP   10.96.177.33   <none>        80/TCP    124m
service/kubernetes        ClusterIP   10.96.0.1      <none>        443/TCP   134m

NAME                                           TOPICS                   BOOTSTRAPSERVERS                            READY   REASON   AGE
kafkasource.sources.knative.dev/kafka-source   ["knative-demo-topic"]   ["my-cluster-kafka-bootstrap.kafka:9092"]   True             124m
```

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* use apply, remove generate name

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix channel and source symlinks

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Fix style check

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

* Delete global resources during KafkaSource migration

- Delete mutatingwebhookconfigurations.admissionregistration.k8s.io/defaulting.webhook.kafka.sources.knative.dev
- Delete validatingwebhookconfigurations.admissionregistration.k8s.io/validation.webhook.kafka.sources.knative.dev
- Delete validatingwebhookconfigurations.admissionregistration.k8s.io/config.webhook.kafka.sources.knative.dev
- Delete service knative-eventing/kafka-source-webhook
- Delete Service knative-eventing/kafka-controller
- Delete deployment.apps/kafka-controller-manager
- Delete ClusterRoleBinding/eventing-sources-kafka-controller
- Delete ClusterRoleBinding/eventing-sources-kafka-controller-addressable-resolver
- Delete ClusterRole/eventing-sources-kafka-controller
- Delete ServiceAccount knative-eventing/kafka-controller-manager
- Delete ConfigMap knative-eventing/config-kafka-source-defaults
- Delete Secret knative-eventing/kafka-source-webhook-certs

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
@pierDipi pierDipi restored the kafka-source-migration-post-install branch August 5, 2024 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane area/test lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants