[Bug]: After concurrent upsert, scanning binlog and found more 14046 data than expected #43574
Description
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: master-20250725-d23205b7-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): woodpecker
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:Current Behavior
server
- mq: woodpecker
- config
common:
enabledJSONKeyStats: true
storage:
enablev2: false
dataCoord:
compaction:
clustering:
autoEnable: true
enableActiveStandby: true
indexCoord:
enableActiveStandby: true
log:
level: debug
queryCoord:
enableActiveStandby: true
queryNode:
enableSegmentPrune: true
rootCoord:
enableActiveStandby: true
client tests
- create a collection with schema (enable partition-key):
{'auto_id': False,
'description': '',
'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}},
{'name': 'int64_1', 'description': '', 'type': <DataType.INT64: 5>, 'is_partition_key': True}, {'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>}],
'enable_dynamic_field': False} (base.py:329)
- create vector index -> insert 20m entities -> flush -> index again -> load
- concurrent requests: upsert + flush + query + search
results
- 14046 more entities
Expected Behavior
No response
Steps To Reproduce
Milvus Log
- argo: https://argo-workflows.zilliz.cc/archived-workflows/qa/803a4134-76b3-41ac-ba08-e4396ada5f5a?nodeId=zong-sn-stable-0725-2414775611
- pods:
After my confirmation, all nodes restarted due to disconnected from etcd
sn-par-key-op-7-1978-etcd-0 1/1 Running 0 8h 10.104.27.92 4am-node31 <none> <none>
sn-par-key-op-7-1978-etcd-1 1/1 Running 0 8h 10.104.19.91 4am-node28 <none> <none>
sn-par-key-op-7-1978-etcd-2 1/1 Running 1 (6h32m ago) 8h 10.104.30.132 4am-node38 <none> <none>
sn-par-key-op-7-1978-milvus-datanode-6bf54cdb6f-l69ds 1/1 Running 0 8h 10.104.27.96 4am-node31 <none> <none>
sn-par-key-op-7-1978-milvus-datanode-6bf54cdb6f-n5hfb 1/1 Running 0 8h 10.104.23.74 4am-node27 <none> <none>
sn-par-key-op-7-1978-milvus-datanode-6bf54cdb6f-pjcb4 1/1 Running 0 8h 10.104.19.97 4am-node28 <none> <none>
sn-par-key-op-7-1978-milvus-mixcoord-6cbb585cbd-6xbts 1/1 Running 1 (6h49m ago) 8h 10.104.6.11 4am-node13 <none> <none>
sn-par-key-op-7-1978-milvus-proxy-7c867df9d7-69zdj 1/1 Running 1 (6h49m ago) 8h 10.104.27.97 4am-node31 <none> <none>
sn-par-key-op-7-1978-milvus-querynode-0-679b8765c4-7jlbk 1/1 Running 0 8h 10.104.6.12 4am-node13 <none> <none>
sn-par-key-op-7-1978-milvus-querynode-0-679b8765c4-cwtrw 1/1 Running 1 (6h49m ago) 8h 10.104.23.75 4am-node27 <none> <none>
sn-par-key-op-7-1978-milvus-querynode-0-679b8765c4-fqbn9 1/1 Running 0 8h 10.104.27.98 4am-node31 <none> <none>
sn-par-key-op-7-1978-milvus-streamingnode-6df7694f67-6xxld 1/1 Running 1 (6h49m ago) 8h 10.104.21.152 4am-node24 <none> <none>
sn-par-key-op-7-1978-milvus-streamingnode-6df7694f67-7d8h4 1/1 Running 2 (6h47m ago) 8h 10.104.19.98 4am-node28 <none> <none>
sn-par-key-op-7-1978-minio-0 1/1 Running 0 8h 10.104.27.93 4am-node31 <none> <none>
sn-par-key-op-7-1978-minio-1 1/1 Running 0 8h 10.104.19.92 4am-node28 <none> <none>
sn-par-key-op-7-1978-minio-2 1/1 Running 0 8h 10.104.30.133 4am-node38 <none> <none>
sn-par-key-op-7-1978-minio-3 1/1 Running 0 8h 10.104.21.151 4am-node24 <none> <none>
Anything else?
No response