这是indexloc提供的服务,不要输入任何密码
Skip to content

[Enhancement] Add configurable commit triggers based on data volume/count in Kafka Connect #13633

@juruoO-o

Description

@juruoO-o

Proposed Change

Problem Description

Currently, Iceberg Kafka Connect only commits based on a fixed time interval (300,000ms by default). This can lead to:

  1. Delayed commits when data volume is low
  2. Memory pressure when data volume is high
  3. Suboptimal performance in varying workload scenarios

Proposed Enhancement

Add additional commit triggers similar to Kafka's approach:

Current Behavior

private static final int COMMIT_INTERVAL_MS_DEFAULT = 300_000; // Only time-based


### Proposal document

_No response_

### Specifications

- [ ] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [x] Other

Metadata

Metadata

Assignees

No one assigned

    Labels

    proposalIceberg Improvement Proposal (spec/major changes/etc)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions