kafka-connect auto-create not setting identifier-field-ids #13623
Description
Apache Iceberg version
1.9.2 (latest release)
Query engine
Kafka Connect
Please describe the bug 🐞
I'm trying out the kafka connect plugin to write messages to an iceberg table
my connector config
{
"name": "test-sink",
"config": {
"tasks.max": "1",
"connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
"topics": "test-topic",
"iceberg.tables.auto-create-enabled": "true",
"iceberg.tables.evolve-schema-enabled": "true",
"iceberg.tables": "default.test_table",
"iceberg.tables.default-id-columns": "id",
"iceberg.tables.default-partition-by": "partition_id",
"iceberg.catalog.type": "rest",
"iceberg.catalog.client.region": "us-east-1",
"iceberg.catalog.uri": "http://iceberg:8181",
"iceberg.catalog.warehouse": "s3://bucket/warehouse/",
"iceberg.catalog.s3.secret-access-key": "minioadmin",
"iceberg.catalog.s3.endpoint": "http://minio:9000",
"iceberg.catalog.s3.path-style-access": "true",
"iceberg.catalog.s3.access-key-id": "minioadmin",
"iceberg.control.commit.interval-ms": 10000
}
}I'm using the docker-compose in the runtime folder, which has been working great. However I noticed that setting iceberg.tables.auto-create-enabled to true and also specify an identifier column in iceberg.tables.default-id-columns, it doesn't add the identifier-field-ids property to the schema when it creates the table on that first commit.
This ends up with the connector duplicating the same message, despite specifying the identifier field.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time