这是indexloc提供的服务,不要输入任何密码
Skip to content

Expose JVM and Kafka clients metrics #435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

pierDipi
Copy link
Member

To better understand our data plane components state, we can
export Kafka clients metrics and JVM metrics (JVM metrics are
disabled by default).

Proposed Changes

  • Expose Kafka clients metrics
  • Expose JVM metrics (disabled by default)

Release Note

:gift: Add new feature
- `kafka-broker-receiver`, `kafka-broker-dispatcher`, and `kafka-sink-receiver` expose Kafka client metrics.
- `kafka-broker-receiver`, `kafka-broker-dispatcher`, and `kafka-sink-receiver` expose JVM metrics (disabled by default)

/kind enhancement

Example metrics

# HELP kafka_consumer_coordinator_failed_rebalance_rate_per_hour The number of failed rebalance events per hour
# TYPE kafka_consumer_coordinator_failed_rebalance_rate_per_hour gauge
kafka_consumer_coordinator_failed_rebalance_rate_per_hour{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_request_rate The number of requests sent per second
# TYPE kafka_consumer_request_rate gauge
kafka_consumer_request_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 6.417842512517069
# HELP vertx_eventbus_handlers Number of event bus handlers in use
# TYPE vertx_eventbus_handlers gauge
vertx_eventbus_handlers 1.0
# HELP kafka_producer_batch_size_avg The average number of bytes sent per partition per-request.
# TYPE kafka_producer_batch_size_avg gauge
kafka_producer_batch_size_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_connection_close_rate The number of connections closed per second
# TYPE kafka_consumer_connection_close_rate gauge
kafka_consumer_connection_close_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_fetch_manager_fetch_throttle_time_avg The average throttle time in ms
# TYPE kafka_consumer_fetch_manager_fetch_throttle_time_avg gauge
kafka_consumer_fetch_manager_fetch_throttle_time_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_waiting_threads The number of user threads blocked waiting for buffer memory to enqueue their records
# TYPE kafka_producer_waiting_threads gauge
kafka_producer_waiting_threads{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_coordinator_rebalance_rate_per_hour The number of successful rebalance events per hour, each event is composed of several failed re-trials until it succeeded
# TYPE kafka_consumer_coordinator_rebalance_rate_per_hour gauge
kafka_consumer_coordinator_rebalance_rate_per_hour{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_io_wait_ratio The fraction of time the I/O thread spent waiting
# TYPE kafka_consumer_io_wait_ratio gauge
kafka_consumer_io_wait_ratio{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.9926071794679229
# HELP kafka_consumer_coordinator_rebalance_latency_total The total number of milliseconds this consumer has spent in successful rebalances since creation
# TYPE kafka_consumer_coordinator_rebalance_latency_total counter
kafka_consumer_coordinator_rebalance_latency_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 283.0
# HELP kafka_consumer_coordinator_last_rebalance_seconds_ago The number of seconds since the last successful rebalance event
# TYPE kafka_consumer_coordinator_last_rebalance_seconds_ago gauge
kafka_consumer_coordinator_last_rebalance_seconds_ago{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 7569.0
# HELP kafka_consumer_network_io_rate The number of network operations (reads or writes) on all connections per second
# TYPE kafka_consumer_network_io_rate gauge
kafka_consumer_network_io_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 12.994697435197196
# HELP kafka_consumer_coordinator_join_total The total number of group joins
# TYPE kafka_consumer_coordinator_join_total counter
kafka_consumer_coordinator_join_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 2.0
# HELP kafka_producer_produce_throttle_time_max The maximum time in ms a request was throttled by a broker
# TYPE kafka_producer_produce_throttle_time_max gauge
kafka_producer_produce_throttle_time_max{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_fetch_manager_bytes_consumed_rate The average number of bytes consumed per second for a topic
# TYPE kafka_consumer_fetch_manager_bytes_consumed_rate gauge
kafka_consumer_fetch_manager_bytes_consumed_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",topic="knative-broker-default-my-kafka-broker",} 24.30564037423358
# HELP kafka_producer_buffer_available_bytes The total amount of buffer memory that is not being used (either unallocated or in the free list).
# TYPE kafka_producer_buffer_available_bytes gauge
kafka_producer_buffer_available_bytes{client_id="producer-1",kafka_version="2.6.0",} 3.3554432E7
# HELP kafka_producer_record_size_max The maximum record size
# TYPE kafka_producer_record_size_max gauge
kafka_producer_record_size_max{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_request_latency_avg The average request latency in ms
# TYPE kafka_producer_request_latency_avg gauge
kafka_producer_request_latency_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_request_size_max The maximum size of any request sent.
# TYPE kafka_producer_request_size_max gauge
kafka_producer_request_size_max{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_node_request_rate The number of requests sent per second
# TYPE kafka_producer_node_request_rate gauge
kafka_producer_node_request_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_producer_node_request_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 0.0
# HELP kafka_producer_connection_close_rate The number of connections closed per second
# TYPE kafka_producer_connection_close_rate gauge
kafka_producer_connection_close_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_response_total The total number of responses received
# TYPE kafka_consumer_response_total counter
kafka_consumer_response_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 48424.0
# HELP vertx_http_client_response_time_seconds_max Response time
# TYPE vertx_http_client_response_time_seconds_max gauge
vertx_http_client_response_time_seconds_max{code="202",method="POST",} 0.016847085
# HELP vertx_http_client_response_time_seconds Response time
# TYPE vertx_http_client_response_time_seconds summary
vertx_http_client_response_time_seconds_count{code="202",method="POST",} 378.0
vertx_http_client_response_time_seconds_sum{code="202",method="POST",} 5.419254652
# HELP kafka_app_info_start_time_ms Metric indicating start-time-ms
# TYPE kafka_app_info_start_time_ms gauge
kafka_app_info_start_time_ms{client_id="producer-1",kafka_version="2.6.0",} 1.606328965281E12
kafka_app_info_start_time_ms{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1.606328965193E12
# HELP kafka_consumer_time_between_poll_max The max delay between invocations of poll().
# TYPE kafka_consumer_time_between_poll_max gauge
kafka_consumer_time_between_poll_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1002.0
# HELP kafka_consumer_fetch_manager_fetch_size_avg The average number of bytes fetched per request for a topic
# TYPE kafka_consumer_fetch_manager_fetch_size_avg gauge
kafka_consumer_fetch_manager_fetch_size_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",topic="knative-broker-default-my-kafka-broker",} 331.6666666666667
# HELP kafka_consumer_successful_reauthentication_total The total number of successful re-authentication of connections
# TYPE kafka_consumer_successful_reauthentication_total counter
kafka_consumer_successful_reauthentication_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_connection_creation_rate The number of new connections established per second
# TYPE kafka_producer_connection_creation_rate gauge
kafka_producer_connection_creation_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_fetch_manager_records_consumed_rate The average number of records consumed per second for a topic
# TYPE kafka_consumer_fetch_manager_records_consumed_rate gauge
kafka_consumer_fetch_manager_records_consumed_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",topic="knative-broker-default-my-kafka-broker",} 0.07328333781175954
# HELP kafka_consumer_node_incoming_byte_rate The number of incoming bytes per second
# TYPE kafka_consumer_node_incoming_byte_rate gauge
kafka_consumer_node_incoming_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 44.12435044215517
kafka_consumer_node_incoming_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 43.887147335423194
kafka_consumer_node_incoming_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 44.29997411040474
kafka_consumer_node_incoming_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_consumer_node_incoming_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 5.4830631343086855
# HELP kafka_producer_response_total The total number of responses received
# TYPE kafka_producer_response_total counter
kafka_producer_response_total{client_id="producer-1",kafka_version="2.6.0",} 28.0
# HELP kafka_consumer_node_request_rate The number of requests sent per second
# TYPE kafka_consumer_node_request_rate gauge
kafka_consumer_node_request_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 2.0058632927017435
kafka_consumer_node_request_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 1.9948703334283269
kafka_consumer_node_request_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_consumer_node_request_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 2.013693113169553
kafka_consumer_node_request_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 0.4096681687832855
# HELP vertx_eventbus_processed_total Number of processed messages
# TYPE vertx_eventbus_processed_total counter
vertx_eventbus_processed_total{side="local",} 1.0
# HELP kafka_producer_buffer_total_bytes The maximum amount of buffer memory the client can use (whether or not it is currently used).
# TYPE kafka_producer_buffer_total_bytes gauge
kafka_producer_buffer_total_bytes{client_id="producer-1",kafka_version="2.6.0",} 3.3554432E7
# HELP kafka_consumer_node_request_size_max The maximum size of any request sent.
# TYPE kafka_consumer_node_request_size_max gauge
kafka_consumer_node_request_size_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 96.0
kafka_consumer_node_request_size_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 96.0
kafka_consumer_node_request_size_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 96.0
kafka_consumer_node_request_size_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_consumer_node_request_size_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 250.0
# HELP kafka_consumer_coordinator_sync_total The total number of group syncs
# TYPE kafka_consumer_coordinator_sync_total counter
kafka_consumer_coordinator_sync_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 2.0
# HELP kafka_producer_record_size_avg The average record size
# TYPE kafka_producer_record_size_avg gauge
kafka_producer_record_size_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_connection_creation_total The total number of new connections established
# TYPE kafka_producer_connection_creation_total counter
kafka_producer_connection_creation_total{client_id="producer-1",kafka_version="2.6.0",} 2.0
# HELP kafka_producer_record_error_rate The average per-second number of record sends that resulted in errors
# TYPE kafka_producer_record_error_rate gauge
kafka_producer_record_error_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_successful_authentication_total The total number of connections with successful authentication
# TYPE kafka_producer_successful_authentication_total counter
kafka_producer_successful_authentication_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_fetch_manager_fetch_latency_max The max time taken for any fetch request.
# TYPE kafka_consumer_fetch_manager_fetch_latency_max gauge
kafka_consumer_fetch_manager_fetch_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 517.0
# HELP kafka_producer_node_request_total The total number of requests sent
# TYPE kafka_producer_node_request_total counter
kafka_producer_node_request_total{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 2.0
kafka_producer_node_request_total{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 26.0
# HELP kafka_consumer_fetch_manager_records_lag_max The max lag of the partition
# TYPE kafka_consumer_fetch_manager_records_lag_max gauge
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="6",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="5",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="8",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="7",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="2",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="1",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="4",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="3",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="0",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="9",topic="knative-broker-default-my-kafka-broker",} NaN
# HELP kafka_consumer_coordinator_rebalance_latency_max The max time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded
# TYPE kafka_consumer_coordinator_rebalance_latency_max gauge
kafka_consumer_coordinator_rebalance_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_successful_reauthentication_rate The number of successful re-authentication of connections per second
# TYPE kafka_producer_successful_reauthentication_rate gauge
kafka_producer_successful_reauthentication_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_outgoing_byte_rate The number of outgoing bytes sent to all servers per second
# TYPE kafka_producer_outgoing_byte_rate gauge
kafka_producer_outgoing_byte_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_successful_authentication_total The total number of connections with successful authentication
# TYPE kafka_consumer_successful_authentication_total counter
kafka_consumer_successful_authentication_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_request_total The total number of requests sent
# TYPE kafka_consumer_request_total counter
kafka_consumer_request_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 48427.0
# HELP kafka_producer_incoming_byte_rate The number of bytes read off all sockets per second
# TYPE kafka_producer_incoming_byte_rate gauge
kafka_producer_incoming_byte_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_fetch_manager_records_consumed_total The total number of records consumed for a topic
# TYPE kafka_consumer_fetch_manager_records_consumed_total counter
kafka_consumer_fetch_manager_records_consumed_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",topic="knative-broker-default-my-kafka-broker",} 378.0
# HELP kafka_producer_node_outgoing_byte_total The total number of outgoing bytes
# TYPE kafka_producer_node_outgoing_byte_total counter
kafka_producer_node_outgoing_byte_total{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 80.0
kafka_producer_node_outgoing_byte_total{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 800.0
# HELP kafka_consumer_coordinator_rebalance_total The total number of successful rebalance events, each event is composed of several failed re-trials until it succeeded
# TYPE kafka_consumer_coordinator_rebalance_total counter
kafka_consumer_coordinator_rebalance_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 2.0
# HELP kafka_producer_request_total The total number of requests sent
# TYPE kafka_producer_request_total counter
kafka_producer_request_total{client_id="producer-1",kafka_version="2.6.0",} 28.0
# HELP kafka_consumer_coordinator_heartbeat_response_time_max The max time taken to receive a response to a heartbeat request
# TYPE kafka_consumer_coordinator_heartbeat_response_time_max gauge
kafka_consumer_coordinator_heartbeat_response_time_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 57.0
# HELP vertx_http_client_active_connections Number of connections to the remote host currently opened
# TYPE vertx_http_client_active_connections gauge
vertx_http_client_active_connections 2.0
# HELP kafka_producer_record_send_total The total number of records sent.
# TYPE kafka_producer_record_send_total counter
kafka_producer_record_send_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_batch_split_rate The average number of batch splits per second
# TYPE kafka_producer_batch_split_rate gauge
kafka_producer_batch_split_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_outgoing_byte_rate The number of outgoing bytes sent to all servers per second
# TYPE kafka_consumer_outgoing_byte_rate gauge
kafka_consumer_outgoing_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 663.6171229603332
# HELP kafka_producer_node_request_size_avg The average size of requests sent.
# TYPE kafka_producer_node_request_size_avg gauge
kafka_producer_node_request_size_avg{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_producer_node_request_size_avg{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} NaN
# HELP vertx_http_client_bytes_written_total Number of bytes sent to the remote host
# TYPE vertx_http_client_bytes_written_total counter
vertx_http_client_bytes_written_total 10710.0
# HELP kafka_producer_request_latency_max The maximum request latency in ms
# TYPE kafka_producer_request_latency_max gauge
kafka_producer_request_latency_max{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_connection_creation_rate The number of new connections established per second
# TYPE kafka_consumer_connection_creation_rate gauge
kafka_consumer_connection_creation_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_connection_close_total The total number of connections closed
# TYPE kafka_producer_connection_close_total counter
kafka_producer_connection_close_total{client_id="producer-1",kafka_version="2.6.0",} 1.0
# HELP kafka_consumer_fetch_manager_records_per_request_avg The average number of records in each request for a topic
# TYPE kafka_consumer_fetch_manager_records_per_request_avg gauge
kafka_consumer_fetch_manager_records_per_request_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",topic="knative-broker-default-my-kafka-broker",} 1.0
# HELP kafka_consumer_io_wait_time_ns_avg The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.
# TYPE kafka_consumer_io_wait_time_ns_avg gauge
kafka_consumer_io_wait_time_ns_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 4.814144820419426E7
# HELP vertx_http_client_active_requests Number of requests waiting for a response
# TYPE vertx_http_client_active_requests gauge
vertx_http_client_active_requests{method="POST",} 0.0
# HELP kafka_consumer_successful_reauthentication_rate The number of successful re-authentication of connections per second
# TYPE kafka_consumer_successful_reauthentication_rate gauge
kafka_consumer_successful_reauthentication_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_compression_rate_avg The average compression rate of record batches.
# TYPE kafka_producer_compression_rate_avg gauge
kafka_producer_compression_rate_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_batch_split_total The total number of batch splits
# TYPE kafka_producer_batch_split_total counter
kafka_producer_batch_split_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_fetch_manager_records_lead_avg The average lead of the partition
# TYPE kafka_consumer_fetch_manager_records_lead_avg gauge
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="6",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="5",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="8",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="7",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="2",topic="knative-broker-default-my-kafka-broker",} 46.0
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="1",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="4",topic="knative-broker-default-my-kafka-broker",} 54.5
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="3",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="0",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="9",topic="knative-broker-default-my-kafka-broker",} NaN
# HELP kafka_consumer_poll_idle_ratio_avg The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records.
# TYPE kafka_consumer_poll_idle_ratio_avg gauge
kafka_consumer_poll_idle_ratio_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.49995397695231986
# HELP kafka_consumer_failed_reauthentication_total The total number of failed re-authentication of connections
# TYPE kafka_consumer_failed_reauthentication_total counter
kafka_consumer_failed_reauthentication_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_successful_authentication_no_reauth_total The total number of connections with successful authentication where the client does not support re-authentication
# TYPE kafka_producer_successful_authentication_no_reauth_total counter
kafka_producer_successful_authentication_no_reauth_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_bufferpool_wait_time_total The total time an appender waits for space allocation.
# TYPE kafka_producer_bufferpool_wait_time_total counter
kafka_producer_bufferpool_wait_time_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_request_size_avg The average size of requests sent.
# TYPE kafka_consumer_request_size_avg gauge
kafka_consumer_request_size_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 103.40425531914893
# HELP kafka_consumer_fetch_manager_preferred_read_replica The current read replica for the partition, or -1 if reading from leader
# TYPE kafka_consumer_fetch_manager_preferred_read_replica gauge
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="6",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="5",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="8",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="7",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="2",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="1",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="4",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="3",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="0",topic="knative-broker-default-my-kafka-broker",} -1.0
kafka_consumer_fetch_manager_preferred_read_replica{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="9",topic="knative-broker-default-my-kafka-broker",} -1.0
# HELP kafka_consumer_coordinator_last_heartbeat_seconds_ago The number of seconds since the last coordinator heartbeat was sent
# TYPE kafka_consumer_coordinator_last_heartbeat_seconds_ago gauge
kafka_consumer_coordinator_last_heartbeat_seconds_ago{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1.0
# HELP kafka_consumer_coordinator_rebalance_latency_avg The average time taken for a group to complete a successful rebalance, which may be composed of several failed re-trials until it succeeded
# TYPE kafka_consumer_coordinator_rebalance_latency_avg gauge
kafka_consumer_coordinator_rebalance_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_connection_creation_total The total number of new connections established
# TYPE kafka_consumer_connection_creation_total counter
kafka_consumer_connection_creation_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 5.0
# HELP vertx_http_client_requests_total Number of requests sent
# TYPE vertx_http_client_requests_total counter
vertx_http_client_requests_total{method="POST",} 378.0
# HELP kafka_producer_requests_in_flight The current number of in-flight requests awaiting a response.
# TYPE kafka_producer_requests_in_flight gauge
kafka_producer_requests_in_flight{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_coordinator_join_time_max The max time taken for a group rejoin
# TYPE kafka_consumer_coordinator_join_time_max gauge
kafka_consumer_coordinator_join_time_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_node_incoming_byte_total The total number of incoming bytes
# TYPE kafka_producer_node_incoming_byte_total counter
kafka_producer_node_incoming_byte_total{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 605.0
kafka_producer_node_incoming_byte_total{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 6341.0
# HELP kafka_producer_select_total The total number of times the I/O layer checked for new I/O to perform
# TYPE kafka_producer_select_total counter
kafka_producer_select_total{client_id="producer-1",kafka_version="2.6.0",} 311.0
# HELP kafka_consumer_node_request_size_avg The average size of requests sent.
# TYPE kafka_consumer_node_request_size_avg gauge
kafka_consumer_node_request_size_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 96.0
kafka_consumer_node_request_size_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 96.0
kafka_consumer_node_request_size_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_consumer_node_request_size_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 96.0
kafka_consumer_node_request_size_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 200.0
# HELP kafka_producer_record_retry_total The total number of retried record sends
# TYPE kafka_producer_record_retry_total counter
kafka_producer_record_retry_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_node_incoming_byte_rate The number of incoming bytes per second
# TYPE kafka_producer_node_incoming_byte_rate gauge
kafka_producer_node_incoming_byte_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_producer_node_incoming_byte_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 0.0
# HELP kafka_consumer_failed_reauthentication_rate The number of failed re-authentication of connections per second
# TYPE kafka_consumer_failed_reauthentication_rate gauge
kafka_consumer_failed_reauthentication_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP vertx_http_client_response_bytes_max Size of responses in bytes
# TYPE vertx_http_client_response_bytes_max gauge
vertx_http_client_response_bytes_max{code="202",method="POST",} 0.0
# HELP vertx_http_client_response_bytes Size of responses in bytes
# TYPE vertx_http_client_response_bytes summary
vertx_http_client_response_bytes_count{code="202",method="POST",} 378.0
vertx_http_client_response_bytes_sum{code="202",method="POST",} 0.0
# HELP kafka_consumer_node_request_total The total number of requests sent
# TYPE kafka_consumer_node_request_total counter
kafka_consumer_node_request_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 15191.0
kafka_consumer_node_request_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 15165.0
kafka_consumer_node_request_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 15158.0
kafka_consumer_node_request_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 3.0
kafka_consumer_node_request_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 2910.0
# HELP kafka_consumer_failed_authentication_total The total number of connections with failed authentication
# TYPE kafka_consumer_failed_authentication_total counter
kafka_consumer_failed_authentication_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_request_size_max The maximum size of any request sent.
# TYPE kafka_consumer_request_size_max gauge
kafka_consumer_request_size_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 250.0
# HELP kafka_consumer_coordinator_sync_time_avg The average time taken for a group sync
# TYPE kafka_consumer_coordinator_sync_time_avg gauge
kafka_consumer_coordinator_sync_time_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP vertx_eventbus_published_total Number of messages published (publish / subscribe)
# TYPE vertx_eventbus_published_total counter
vertx_eventbus_published_total{side="local",} 3789.0
# HELP kafka_consumer_node_request_latency_max
# TYPE kafka_consumer_node_request_latency_max gauge
kafka_consumer_node_request_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} NaN
kafka_consumer_node_request_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} NaN
kafka_consumer_node_request_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} NaN
kafka_consumer_node_request_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_consumer_node_request_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} NaN
# HELP kafka_consumer_fetch_manager_records_lag_avg The average lag of the partition
# TYPE kafka_consumer_fetch_manager_records_lag_avg gauge
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="6",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="5",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="8",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="7",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="2",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="1",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="4",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="3",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="0",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lag_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="9",topic="knative-broker-default-my-kafka-broker",} NaN
# HELP vertx_http_client_queue_time_seconds_max Time spent in queue before being processed
# TYPE vertx_http_client_queue_time_seconds_max gauge
vertx_http_client_queue_time_seconds_max 5.11038E-4
# HELP vertx_http_client_queue_time_seconds Time spent in queue before being processed
# TYPE vertx_http_client_queue_time_seconds summary
vertx_http_client_queue_time_seconds_count 378.0
vertx_http_client_queue_time_seconds_sum 0.450283051
# HELP kafka_producer_node_response_total The total number of responses received
# TYPE kafka_producer_node_response_total counter
kafka_producer_node_response_total{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 2.0
kafka_producer_node_response_total{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 26.0
# HELP kafka_consumer_coordinator_heartbeat_total The total number of heartbeats
# TYPE kafka_consumer_coordinator_heartbeat_total counter
kafka_consumer_coordinator_heartbeat_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 2524.0
# HELP kafka_producer_bufferpool_wait_ratio The fraction of time an appender waits for space allocation.
# TYPE kafka_producer_bufferpool_wait_ratio gauge
kafka_producer_bufferpool_wait_ratio{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_successful_authentication_no_reauth_total The total number of connections with successful authentication where the client does not support re-authentication
# TYPE kafka_consumer_successful_authentication_no_reauth_total counter
kafka_consumer_successful_authentication_no_reauth_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_select_rate The number of times the I/O layer checked for new I/O to perform per second
# TYPE kafka_producer_select_rate gauge
kafka_producer_select_rate{client_id="producer-1",kafka_version="2.6.0",} 0.044896401553415496
# HELP kafka_consumer_connection_close_total The total number of connections closed
# TYPE kafka_consumer_connection_close_total counter
kafka_consumer_connection_close_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1.0
# HELP kafka_producer_failed_reauthentication_total The total number of failed re-authentication of connections
# TYPE kafka_producer_failed_reauthentication_total counter
kafka_producer_failed_reauthentication_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_coordinator_sync_time_max The max time taken for a group sync
# TYPE kafka_consumer_coordinator_sync_time_max gauge
kafka_consumer_coordinator_sync_time_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_batch_size_max The max number of bytes sent per partition per-request.
# TYPE kafka_producer_batch_size_max gauge
kafka_producer_batch_size_max{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_io_ratio The fraction of time the I/O thread spent doing I/O
# TYPE kafka_consumer_io_ratio gauge
kafka_consumer_io_ratio{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0016542118426142959
# HELP kafka_producer_node_response_rate The number of responses received per second
# TYPE kafka_producer_node_response_rate gauge
kafka_producer_node_response_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_producer_node_response_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 0.0
# HELP kafka_producer_io_wait_ratio The fraction of time the I/O thread spent waiting
# TYPE kafka_producer_io_wait_ratio gauge
kafka_producer_io_wait_ratio{client_id="producer-1",kafka_version="2.6.0",} 1.3482377320358274
# HELP kafka_producer_failed_reauthentication_rate The number of failed re-authentication of connections per second
# TYPE kafka_producer_failed_reauthentication_rate gauge
kafka_producer_failed_reauthentication_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_reauthentication_latency_avg The average latency observed due to re-authentication
# TYPE kafka_producer_reauthentication_latency_avg gauge
kafka_producer_reauthentication_latency_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_response_rate The number of responses received per second
# TYPE kafka_producer_response_rate gauge
kafka_producer_response_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_successful_authentication_rate The number of connections with successful authentication per second
# TYPE kafka_producer_successful_authentication_rate gauge
kafka_producer_successful_authentication_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_coordinator_sync_rate The number of group syncs per second
# TYPE kafka_consumer_coordinator_sync_rate gauge
kafka_consumer_coordinator_sync_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_time_between_poll_avg The average delay between invocations of poll().
# TYPE kafka_consumer_time_between_poll_avg gauge
kafka_consumer_time_between_poll_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1000.4102564102565
# HELP kafka_producer_request_rate The number of requests sent per second
# TYPE kafka_producer_request_rate gauge
kafka_producer_request_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_fetch_manager_fetch_total The total number of fetch requests.
# TYPE kafka_consumer_fetch_manager_fetch_total counter
kafka_consumer_fetch_manager_fetch_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 45496.0
# HELP kafka_consumer_fetch_manager_fetch_size_max The maximum number of bytes fetched per request for a topic
# TYPE kafka_consumer_fetch_manager_fetch_size_max gauge
kafka_consumer_fetch_manager_fetch_size_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",topic="knative-broker-default-my-kafka-broker",} 335.0
# HELP kafka_consumer_fetch_manager_records_lead The latest lead of the partition
# TYPE kafka_consumer_fetch_manager_records_lead gauge
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="6",topic="knative-broker-default-my-kafka-broker",} 48.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="5",topic="knative-broker-default-my-kafka-broker",} 46.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="8",topic="knative-broker-default-my-kafka-broker",} 29.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="7",topic="knative-broker-default-my-kafka-broker",} 36.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="2",topic="knative-broker-default-my-kafka-broker",} 46.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="1",topic="knative-broker-default-my-kafka-broker",} 25.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="4",topic="knative-broker-default-my-kafka-broker",} 55.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="3",topic="knative-broker-default-my-kafka-broker",} 46.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="0",topic="knative-broker-default-my-kafka-broker",} 42.0
kafka_consumer_fetch_manager_records_lead{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="9",topic="knative-broker-default-my-kafka-broker",} 35.0
# HELP kafka_producer_failed_authentication_rate The number of connections with failed authentication per second
# TYPE kafka_producer_failed_authentication_rate gauge
kafka_producer_failed_authentication_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_coordinator_partition_assigned_latency_avg The average time taken for a partition-assigned rebalance listener callback
# TYPE kafka_consumer_coordinator_partition_assigned_latency_avg gauge
kafka_consumer_coordinator_partition_assigned_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_fetch_manager_bytes_consumed_total The total number of bytes consumed for a topic
# TYPE kafka_consumer_fetch_manager_bytes_consumed_total counter
kafka_consumer_fetch_manager_bytes_consumed_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",topic="knative-broker-default-my-kafka-broker",} 125367.0
# HELP kafka_consumer_reauthentication_latency_max The max latency observed due to re-authentication
# TYPE kafka_consumer_reauthentication_latency_max gauge
kafka_consumer_reauthentication_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_network_io_rate The number of network operations (reads or writes) on all connections per second
# TYPE kafka_producer_network_io_rate gauge
kafka_producer_network_io_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_incoming_byte_total The total number of bytes read off all sockets
# TYPE kafka_producer_incoming_byte_total counter
kafka_producer_incoming_byte_total{client_id="producer-1",kafka_version="2.6.0",} 6946.0
# HELP kafka_consumer_fetch_manager_fetch_latency_avg The average time taken for a fetch request.
# TYPE kafka_consumer_fetch_manager_fetch_latency_avg gauge
kafka_consumer_fetch_manager_fetch_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 498.93884892086334
# HELP kafka_consumer_iotime_total The total time the I/O thread spent doing I/O
# TYPE kafka_consumer_iotime_total counter
kafka_consumer_iotime_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1.2629662201E10
# HELP kafka_consumer_coordinator_failed_rebalance_total The total number of failed rebalance events
# TYPE kafka_consumer_coordinator_failed_rebalance_total counter
kafka_consumer_coordinator_failed_rebalance_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1.0
# HELP vertx_eventbus_received_total Number of messages received
# TYPE vertx_eventbus_received_total counter
vertx_eventbus_received_total{side="local",} 3789.0
# HELP kafka_producer_record_queue_time_avg The average time in ms record batches spent in the send buffer.
# TYPE kafka_producer_record_queue_time_avg gauge
kafka_producer_record_queue_time_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_node_incoming_byte_total The total number of incoming bytes
# TYPE kafka_consumer_node_incoming_byte_total counter
kafka_consumer_node_incoming_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 413415.0
kafka_consumer_node_incoming_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 388803.0
kafka_consumer_node_incoming_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 1156.0
kafka_consumer_node_incoming_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 389746.0
kafka_consumer_node_incoming_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 65875.0
# HELP kafka_producer_node_request_latency_max
# TYPE kafka_producer_node_request_latency_max gauge
kafka_producer_node_request_latency_max{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_producer_node_request_latency_max{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} NaN
# HELP kafka_producer_node_outgoing_byte_rate The number of outgoing bytes per second
# TYPE kafka_producer_node_outgoing_byte_rate gauge
kafka_producer_node_outgoing_byte_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_producer_node_outgoing_byte_rate{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} 0.0
# HELP kafka_producer_record_queue_time_max The maximum time in ms record batches spent in the send buffer.
# TYPE kafka_producer_record_queue_time_max gauge
kafka_producer_record_queue_time_max{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_node_request_size_max The maximum size of any request sent.
# TYPE kafka_producer_node_request_size_max gauge
kafka_producer_node_request_size_max{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_producer_node_request_size_max{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} NaN
# HELP kafka_consumer_select_rate The number of times the I/O layer checked for new I/O to perform per second
# TYPE kafka_consumer_select_rate gauge
kafka_consumer_select_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 20.617618278224064
# HELP kafka_producer_io_time_ns_avg The average length of time for I/O per select call in nanoseconds.
# TYPE kafka_producer_io_time_ns_avg gauge
kafka_producer_io_time_ns_avg{client_id="producer-1",kafka_version="2.6.0",} 8015.0
# HELP kafka_producer_io_waittime_total The total time the I/O thread spent waiting
# TYPE kafka_producer_io_waittime_total counter
kafka_producer_io_waittime_total{client_id="producer-1",kafka_version="2.6.0",} 7.560882876874E12
# HELP kafka_consumer_fetch_manager_records_lag The latest lag of the partition
# TYPE kafka_consumer_fetch_manager_records_lag gauge
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="6",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="5",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="8",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="7",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="2",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="1",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="4",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="3",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="0",topic="knative-broker-default-my-kafka-broker",} 0.0
kafka_consumer_fetch_manager_records_lag{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="9",topic="knative-broker-default-my-kafka-broker",} 0.0
# HELP vertx_eventbus_delivered_total Number of messages delivered to handlers
# TYPE vertx_eventbus_delivered_total counter
vertx_eventbus_delivered_total{side="local",} 1.0
# HELP kafka_producer_network_io_total The total number of network operations (reads or writes) on all connections
# TYPE kafka_producer_network_io_total counter
kafka_producer_network_io_total{client_id="producer-1",kafka_version="2.6.0",} 56.0
# HELP kafka_producer_request_size_avg The average size of requests sent.
# TYPE kafka_producer_request_size_avg gauge
kafka_producer_request_size_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_select_total The total number of times the I/O layer checked for new I/O to perform
# TYPE kafka_consumer_select_total counter
kafka_consumer_select_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 144011.0
# HELP kafka_consumer_response_rate The number of responses received per second
# TYPE kafka_consumer_response_rate gauge
kafka_consumer_response_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 6.427202832819965
# HELP kafka_consumer_coordinator_partition_lost_latency_max The max time taken for a partition-lost rebalance listener callback
# TYPE kafka_consumer_coordinator_partition_lost_latency_max gauge
kafka_consumer_coordinator_partition_lost_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_coordinator_commit_latency_avg The average time taken for a commit request
# TYPE kafka_consumer_coordinator_commit_latency_avg gauge
kafka_consumer_coordinator_commit_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 7.666666666666667
# HELP kafka_consumer_coordinator_join_time_avg The average time taken for a group rejoin
# TYPE kafka_consumer_coordinator_join_time_avg gauge
kafka_consumer_coordinator_join_time_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_coordinator_heartbeat_rate The number of heartbeats per second
# TYPE kafka_consumer_coordinator_heartbeat_rate gauge
kafka_consumer_coordinator_heartbeat_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.34267337955030713
# HELP kafka_producer_record_send_rate The average number of records sent per second.
# TYPE kafka_producer_record_send_rate gauge
kafka_producer_record_send_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_io_wait_time_ns_avg The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.
# TYPE kafka_producer_io_wait_time_ns_avg gauge
kafka_producer_io_wait_time_ns_avg{client_id="producer-1",kafka_version="2.6.0",} 3.00299731245E10
# HELP kafka_producer_node_request_latency_avg
# TYPE kafka_producer_node_request_latency_avg gauge
kafka_producer_node_request_latency_avg{client_id="producer-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_producer_node_request_latency_avg{client_id="producer-1",kafka_version="2.6.0",node_id="node-2",} NaN
# HELP kafka_consumer_io_waittime_total The total time the I/O thread spent waiting
# TYPE kafka_consumer_io_waittime_total counter
kafka_consumer_io_waittime_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 7.536225499173E12
# HELP kafka_consumer_incoming_byte_rate The number of bytes read off all sockets per second
# TYPE kafka_consumer_incoming_byte_rate gauge
kafka_consumer_incoming_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 177.17251582547112
# HELP kafka_consumer_failed_authentication_rate The number of connections with failed authentication per second
# TYPE kafka_consumer_failed_authentication_rate gauge
kafka_consumer_failed_authentication_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_buffer_exhausted_total The total number of record sends that are dropped due to buffer exhaustion
# TYPE kafka_producer_buffer_exhausted_total counter
kafka_producer_buffer_exhausted_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_node_response_total The total number of responses received
# TYPE kafka_consumer_node_response_total counter
kafka_consumer_node_response_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 15190.0
kafka_consumer_node_response_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 15164.0
kafka_consumer_node_response_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 15157.0
kafka_consumer_node_response_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 3.0
kafka_consumer_node_response_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 2910.0
# HELP kafka_consumer_fetch_manager_fetch_throttle_time_max The maximum throttle time in ms
# TYPE kafka_consumer_fetch_manager_fetch_throttle_time_max gauge
kafka_consumer_fetch_manager_fetch_throttle_time_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_producer_successful_reauthentication_total The total number of successful re-authentication of connections
# TYPE kafka_producer_successful_reauthentication_total counter
kafka_producer_successful_reauthentication_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP vertx_http_client_queue_pending Number of pending elements in queue
# TYPE vertx_http_client_queue_pending gauge
vertx_http_client_queue_pending 0.0
# HELP vertx_eventbus_pending Number of messages not processed yet
# TYPE vertx_eventbus_pending gauge
vertx_eventbus_pending{side="local",} 0.0
# HELP kafka_producer_connection_count_total The current number of active connections.
# TYPE kafka_producer_connection_count_total counter
kafka_producer_connection_count_total{client_id="producer-1",kafka_version="2.6.0",} 1.0
# HELP kafka_consumer_outgoing_byte_total The total number of outgoing bytes sent to all servers
# TYPE kafka_consumer_outgoing_byte_total counter
kafka_consumer_outgoing_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 4973232.0
# HELP kafka_consumer_coordinator_assigned_partitions The number of partitions currently assigned to this consumer
# TYPE kafka_consumer_coordinator_assigned_partitions gauge
kafka_consumer_coordinator_assigned_partitions{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 10.0
# HELP kafka_consumer_last_poll_seconds_ago The number of seconds since the last poll() invocation.
# TYPE kafka_consumer_last_poll_seconds_ago gauge
kafka_consumer_last_poll_seconds_ago{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_node_outgoing_byte_total The total number of outgoing bytes
# TYPE kafka_consumer_node_outgoing_byte_total counter
kafka_consumer_node_outgoing_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 1470080.0
kafka_consumer_node_outgoing_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 1463762.0
kafka_consumer_node_outgoing_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 1463295.0
kafka_consumer_node_outgoing_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 295.0
kafka_consumer_node_outgoing_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 575800.0
# HELP kafka_consumer_coordinator_partition_revoked_latency_avg The average time taken for a partition-revoked rebalance listener callback
# TYPE kafka_consumer_coordinator_partition_revoked_latency_avg gauge
kafka_consumer_coordinator_partition_revoked_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_producer_iotime_total The total time the I/O thread spent doing I/O
# TYPE kafka_producer_iotime_total counter
kafka_producer_iotime_total{client_id="producer-1",kafka_version="2.6.0",} 2.5437391E7
# HELP kafka_consumer_coordinator_commit_total The total number of commit calls
# TYPE kafka_consumer_coordinator_commit_total counter
kafka_consumer_coordinator_commit_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 378.0
# HELP kafka_producer_records_per_request_avg The average number of records per request.
# TYPE kafka_producer_records_per_request_avg gauge
kafka_producer_records_per_request_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_coordinator_join_rate The number of group joins per second
# TYPE kafka_consumer_coordinator_join_rate gauge
kafka_consumer_coordinator_join_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_fetch_manager_fetch_rate The number of fetch requests per second.
# TYPE kafka_consumer_fetch_manager_fetch_rate gauge
kafka_consumer_fetch_manager_fetch_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 6.010454805189308
# HELP kafka_consumer_coordinator_commit_latency_max The max time taken for a commit request
# TYPE kafka_consumer_coordinator_commit_latency_max gauge
kafka_consumer_coordinator_commit_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 9.0
# HELP kafka_consumer_node_request_latency_avg
# TYPE kafka_consumer_node_request_latency_avg gauge
kafka_consumer_node_request_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} NaN
kafka_consumer_node_request_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} NaN
kafka_consumer_node_request_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} NaN
kafka_consumer_node_request_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} NaN
kafka_consumer_node_request_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} NaN
# HELP kafka_consumer_coordinator_partition_assigned_latency_max The max time taken for a partition-assigned rebalance listener callback
# TYPE kafka_consumer_coordinator_partition_assigned_latency_max gauge
kafka_consumer_coordinator_partition_assigned_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_incoming_byte_total The total number of bytes read off all sockets
# TYPE kafka_consumer_incoming_byte_total counter
kafka_consumer_incoming_byte_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 1258995.0
# HELP kafka_producer_failed_authentication_total The total number of connections with failed authentication
# TYPE kafka_producer_failed_authentication_total counter
kafka_producer_failed_authentication_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP vertx_http_client_request_bytes_max Size of requests in bytes
# TYPE vertx_http_client_request_bytes_max gauge
vertx_http_client_request_bytes_max{method="POST",} 31.0
# HELP vertx_http_client_request_bytes Size of requests in bytes
# TYPE vertx_http_client_request_bytes summary
vertx_http_client_request_bytes_count{method="POST",} 378.0
vertx_http_client_request_bytes_sum{method="POST",} 10710.0
# HELP kafka_producer_produce_throttle_time_avg The average time in ms a request was throttled by a broker
# TYPE kafka_producer_produce_throttle_time_avg gauge
kafka_producer_produce_throttle_time_avg{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_coordinator_commit_rate The number of commit calls per second
# TYPE kafka_consumer_coordinator_commit_rate gauge
kafka_consumer_coordinator_commit_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.07356547327121138
# HELP kafka_producer_outgoing_byte_total The total number of outgoing bytes sent to all servers
# TYPE kafka_producer_outgoing_byte_total counter
kafka_producer_outgoing_byte_total{client_id="producer-1",kafka_version="2.6.0",} 880.0
# HELP kafka_producer_record_retry_rate The average per-second number of retried record sends
# TYPE kafka_producer_record_retry_rate gauge
kafka_producer_record_retry_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_reauthentication_latency_avg The average latency observed due to re-authentication
# TYPE kafka_consumer_reauthentication_latency_avg gauge
kafka_consumer_reauthentication_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP http_events_sent_total
# TYPE http_events_sent_total counter
http_events_sent_total 378.0
# HELP vertx_http_client_responses_total Response count with codes
# TYPE vertx_http_client_responses_total counter
vertx_http_client_responses_total{code="202",method="POST",} 378.0
# HELP kafka_producer_buffer_exhausted_rate The average per-second number of record sends that are dropped due to buffer exhaustion
# TYPE kafka_producer_buffer_exhausted_rate gauge
kafka_producer_buffer_exhausted_rate{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_successful_authentication_rate The number of connections with successful authentication per second
# TYPE kafka_consumer_successful_authentication_rate gauge
kafka_consumer_successful_authentication_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_coordinator_partition_lost_latency_avg The average time taken for a partition-lost rebalance listener callback
# TYPE kafka_consumer_coordinator_partition_lost_latency_avg gauge
kafka_consumer_coordinator_partition_lost_latency_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_connection_count_total The current number of active connections.
# TYPE kafka_consumer_connection_count_total counter
kafka_consumer_connection_count_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 4.0
# HELP kafka_producer_reauthentication_latency_max The max latency observed due to re-authentication
# TYPE kafka_producer_reauthentication_latency_max gauge
kafka_producer_reauthentication_latency_max{client_id="producer-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_node_response_rate The number of responses received per second
# TYPE kafka_consumer_node_response_rate gauge
kafka_consumer_node_response_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 2.0054694621695535
kafka_consumer_node_response_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 1.9946997976804488
kafka_consumer_node_response_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_consumer_node_response_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 2.01346142783179
kafka_consumer_node_response_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 0.3426643470926248
# HELP kafka_consumer_network_io_total The total number of network operations (reads or writes) on all connections
# TYPE kafka_consumer_network_io_total counter
kafka_consumer_network_io_total{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 98058.0
# HELP kafka_producer_record_error_total The total number of record sends that resulted in errors
# TYPE kafka_producer_record_error_total counter
kafka_producer_record_error_total{client_id="producer-1",kafka_version="2.6.0",} 0.0
# HELP kafka_consumer_io_time_ns_avg The average length of time for I/O per select call in nanoseconds.
# TYPE kafka_consumer_io_time_ns_avg gauge
kafka_consumer_io_time_ns_avg{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} 80232.92604856512
# HELP kafka_producer_io_ratio The fraction of time the I/O thread spent doing I/O
# TYPE kafka_producer_io_ratio gauge
kafka_producer_io_ratio{client_id="producer-1",kafka_version="2.6.0",} 3.598365807668133E-7
# HELP kafka_consumer_coordinator_partition_revoked_latency_max The max time taken for a partition-revoked rebalance listener callback
# TYPE kafka_consumer_coordinator_partition_revoked_latency_max gauge
kafka_consumer_coordinator_partition_revoked_latency_max{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",} NaN
# HELP kafka_consumer_fetch_manager_records_lead_min The min lead of the partition
# TYPE kafka_consumer_fetch_manager_records_lead_min gauge
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="6",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="5",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="8",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="7",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="2",topic="knative-broker-default-my-kafka-broker",} 46.0
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="1",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="4",topic="knative-broker-default-my-kafka-broker",} 54.0
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="3",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="0",topic="knative-broker-default-my-kafka-broker",} NaN
kafka_consumer_fetch_manager_records_lead_min{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",partition="9",topic="knative-broker-default-my-kafka-broker",} NaN
# HELP kafka_producer_metadata_age The age in seconds of the current producer metadata being used.
# TYPE kafka_producer_metadata_age gauge
kafka_producer_metadata_age{client_id="producer-1",kafka_version="2.6.0",} 74.578
# HELP kafka_consumer_node_outgoing_byte_rate The number of outgoing bytes per second
# TYPE kafka_consumer_node_outgoing_byte_rate gauge
kafka_consumer_node_outgoing_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-0",} 192.54505060478894
kafka_consumer_node_outgoing_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2",} 191.4911805773231
kafka_consumer_node_outgoing_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-1",} 193.29785704012656
kafka_consumer_node_outgoing_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node--1",} 0.0
kafka_consumer_node_outgoing_byte_rate{client_id="consumer-7ff7770d-7070-4118-a016-f989e265c93a-1",kafka_version="2.6.0",node_id="node-2147483647",} 81.92803987164606

@google-cla google-cla bot added the cla: yes Indicates the PR's author has signed the CLA. label Nov 26, 2020
@knative-prow-robot knative-prow-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 26, 2020
@pierDipi pierDipi force-pushed the kafka-clients-metrics branch from 738ae11 to 85c402d Compare November 26, 2020 16:46
@codecov
Copy link

codecov bot commented Nov 26, 2020

Codecov Report

Merging #435 (fc106c0) into master (8bf529f) will increase coverage by 0.59%.
The diff coverage is 89.65%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #435      +/-   ##
============================================
+ Coverage     76.19%   76.79%   +0.59%     
- Complexity      260      269       +9     
============================================
  Files            58       59       +1     
  Lines          1907     1969      +62     
  Branches         82       82              
============================================
+ Hits           1453     1512      +59     
- Misses          339      342       +3     
  Partials        115      115              
Flag Coverage Δ Complexity Δ
java-unittests 77.94% <89.65%> (+0.95%) 0.00 <16.00> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ Complexity Δ
...knative/eventing/kafka/broker/dispatcher/Main.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...v/knative/eventing/kafka/broker/receiver/Main.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...ve/eventing/kafka/broker/core/metrics/Metrics.java 89.65% <89.65%> (ø) 6.00 <6.00> (?)
.../eventing/kafka/broker/receiver/RequestMapper.java 95.68% <90.47%> (-1.32%) 16.00 <1.00> (-1.00)
...tive/eventing/kafka/broker/core/utils/BaseEnv.java 100.00% <100.00%> (ø) 9.00 <1.00> (+1.00)
...kafka/broker/dispatcher/ConsumerRecordHandler.java 85.71% <100.00%> (+0.78%) 16.00 <1.00> (ø)
.../kafka/broker/dispatcher/ConsumerRecordSender.java 100.00% <100.00%> (ø) 1.00 <1.00> (?)
...ting/kafka/broker/dispatcher/ConsumerVerticle.java 100.00% <100.00%> (ø) 5.00 <4.00> (+2.00)
...oker/dispatcher/http/HttpConsumerRecordSender.java 60.00% <100.00%> (+2.85%) 7.00 <1.00> (+1.00)
...r/dispatcher/http/HttpConsumerVerticleFactory.java 88.70% <100.00%> (ø) 21.00 <0.00> (-1.00)
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8bf529f...8042ffa. Read the comment docs.

@pierDipi
Copy link
Member Author

/retest

1 similar comment
@pierDipi
Copy link
Member Author

/retest

@pierDipi pierDipi force-pushed the kafka-clients-metrics branch from 85c402d to 73f003f Compare November 27, 2020 09:05
@pierDipi
Copy link
Member Author

Flaky
/retest

@pierDipi
Copy link
Member Author

/hold
/retest

@knative-prow-robot knative-prow-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 27, 2020
@pierDipi
Copy link
Member Author

connection reset, maven
/retest

@skonto
Copy link
Contributor

skonto commented Dec 1, 2020

Hi, the java Kafka client does not provide any metrics eg. via jmx? Do we need any docs here?

@knative-prow-robot knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2020
@pierDipi
Copy link
Member Author

pierDipi commented Dec 1, 2020

Hi @skonto, we're using micrometer-registry-prometheus to export metrics rather than JMX, are you suggesting to use JMX?

Do we need any docs here?

Yes, I'm thinking to link the page: https://kafka.apache.org/documentation/#selector_monitoring, for knowing which metrics we expose, and clarifying that "-" are replaced by "_" since metric names must match [a-zA-Z_:][a-zA-Z0-9_:]*, consumer-specific metrics have the kafka_consumer_ prefix, and producer-specific metrics have the kafka_producer_ prefix.
Do you think it's enough? or that prefixes could be removed?

We already expose metrics configurations for both consumer and producer: https://github.com/knative-sandbox/eventing-kafka-broker/blob/50e92f981fd837b142425965a828f44c56354e93/data-plane/config/100-config-kafka-broker-data-plane.yaml#L45-L47
and this is documented here: https://knative.dev/docs/eventing/broker/kafka-broker/#kafka-producer-and-consumer-configurations

I'll mark this PR as WIP, since I'd like to keep metrics names consistent with other projects in the Kafka area.

@pierDipi pierDipi changed the title Expose JVM and Kafka clients metrics [WIP] Expose JVM and Kafka clients metrics Dec 1, 2020
@knative-prow-robot knative-prow-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 1, 2020
@pierDipi pierDipi force-pushed the kafka-clients-metrics branch from 73f003f to 6ef241b Compare December 1, 2020 17:48
@knative-prow-robot knative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2020
@pierDipi pierDipi force-pushed the kafka-clients-metrics branch 2 times, most recently from 03837e5 to 8e91dec Compare December 2, 2020 10:40
@pierDipi
Copy link
Member Author

pierDipi commented Dec 2, 2020

Worked, retesting since it seems to be unstable:
/test pull-knative-sandbox-eventing-kafka-broker-integration-tests

@pierDipi
Copy link
Member Author

pierDipi commented Dec 2, 2020

Known flake,
/retest

@skonto
Copy link
Contributor

skonto commented Dec 2, 2020

@pierDipi my thinking was to review existing metrics if available (I know you are using something vert.x specific here).
One option would be if java kafka client emits that already you can expose them using https://github.com/prometheus/jmx_exporter (run as a javaagent or sidecar).
I have done this before but maybe it is simpler just to do that manually via vert.x apis. Just food for thought.

@pierDipi
Copy link
Member Author

pierDipi commented Dec 2, 2020

I did a POC using the JMX exporter, using an initContiner to inject the agent: pierDipi@15d1494, and discussing this with @slinkydeveloper, we concluded that having more control over what metrics we expose it's better long term, and we also need to expose custom metrics that are part of the Broker spec, so we need to use the Vert.x API anyway.

For example, I'm disabling VertxPoolMetrics due to an impressive allocation rate:

image

Also, he suggested that an agent might be heavyweight from a resource perspective.

WDYT?

I have done this before but maybe it is simpler just to do that manually via vert.x apis. Just food for thought.

Thanks, I really appreciate it.

@pierDipi
Copy link
Member Author

pierDipi commented Dec 2, 2020

/test pull-knative-sandbox-eventing-kafka-broker-integration-tests

@pierDipi pierDipi changed the title [WIP] Expose JVM and Kafka clients metrics Expose JVM and Kafka clients metrics Dec 11, 2020
@knative-prow-robot knative-prow-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 11, 2020
@pierDipi
Copy link
Member Author

This PR is ready for reviews.

/cc @slinkydeveloper @skonto

@knative-prow-robot
Copy link
Contributor

@pierDipi: GitHub didn't allow me to request PR reviews from the following users: skonto.

Note that only knative-sandbox members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

This PR is ready for reviews.

/cc @slinkydeveloper @skonto

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
@pierDipi pierDipi force-pushed the kafka-clients-metrics branch from 4aaed10 to 8042ffa Compare December 11, 2020 14:41
@pierDipi
Copy link
Member Author

Rebased.

Copy link
Contributor

@slinkydeveloper slinkydeveloper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I just left a minor comment

@slinkydeveloper
Copy link
Contributor

/lgtm
/approve

@knative-prow-robot knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 14, 2020
@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pierDipi, slinkydeveloper

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [pierDipi,slinkydeveloper]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pierDipi
Copy link
Member Author

/unhold

@knative-prow-robot knative-prow-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 14, 2020
@pierDipi
Copy link
Member Author

Flaky :(

/retest

@pierDipi
Copy link
Member Author

/retest

1 similar comment
@pierDipi
Copy link
Member Author

/retest

@knative-prow-robot knative-prow-robot merged commit 17e3199 into knative-extensions:master Dec 14, 2020
pierDipi added a commit to pierDipi/eventing-kafka-broker that referenced this pull request Feb 27, 2021
* Expose Kafka clients metrics

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Reduce frequency in sacura job

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Keep running sacura if tests fail

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Too many logs in debug mode

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Increase timeout

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Close KafkaClientMetrics binder, WebClient, Producer and Consumer

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Revert sacura changes

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Remove tracing

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Producer metrics on receiver

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>

* Add more tests and docs

Signed-off-by: Pierangelo Di Pilato <pierangelodipilato@gmail.com>
@pierDipi pierDipi deleted the kafka-clients-metrics branch April 14, 2021 21:18
pierDipi added a commit to pierDipi/eventing-kafka-broker that referenced this pull request Oct 21, 2022
…o slow (knative-extensions#422) (knative-extensions#435)

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane area/data-plane area/test cla: yes Indicates the PR's author has signed the CLA. kind/enhancement lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants