This stack provides a complete Apache Kafka development and monitoring environment, built using Confluent Platform components and enhanced with Kpow for enterprise-grade observability and management. It includes a 3-node Kafka cluster, Zookeeper, Schema Registry, Kafka Connect, and Kpow itself.

This architecture is designed for developers and operations teams who need a robust, local Kafka ecosystem for building, testing, and managing Kafka-based applications. It features high availability (3 brokers), schema management for data governance, a data integration framework (Kafka Connect), and a powerful UI (Kpow) for monitoring cluster health, inspecting data, managing topics, and troubleshooting.

It's ideal for scenarios involving event-driven architectures, microservices communication, data integration pipelines, and situations requiring deep visibility into Kafka internals.

• Container: kpow from (factorhouse/kpow:latest (enterprise)) or kpow-ce from (factorhouse/kpow-ce:latest (community))
• An engineering toolkit providing a rich web UI for comprehensive Kafka monitoring, management, and data inspection. Kpow gathers Kafka resource information, stores it locally in internal topics, and delivers custom telemetry and insights. Features include:
  • Comprehensive Real-time Kafka Resource Monitoring:
    • Instant visibility ("X-Ray vision") of brokers, topics, consumer groups, partitions, offsets, and more.
    • Gathers data every minute, with a "Live mode" for real-time updates.
    • No JMX access required.
  • Advanced Consumer and Streams Monitoring (Compute Console):
    • Visualize message throughput and lag for consumers (and soon, Kafka Streams topologies).
    • Multi-dimensional consumer lag insights from custom derived telemetry.
    • Ability to reset consumption at group, host, member, topic, or assignment level.
  • Deep Data Inspection with kJQ:
    • Powerful JQ-like querying (kJQ) to search tens of thousands of messages per second.
    • Supports JSON, Apache Avro®, Transit, EDN, and Protobuf messages (with custom serdes for Protobuf to JSON).
  • Schema Registry Integration: Full support for controlling and monitoring Schema Registries.
  • Kafka Connect Cluster Management: Full support for controlling and monitoring Kafka Connect clusters.
  • Enterprise-Grade Security & Governance:
    • Authentication: Supports DB, File, LDAP, SAML, or OpenID configurations.
    • Authorization: Simple or Role-Based Access Controls (RBAC). The original summary also mentioned JAAS, often paired with RBAC and configured via volume mounts (Enterprise edition (ee) only).
    • Data Policies: Includes capabilities for masking and redaction of sensitive data (e.g., PII, Credit Card).
    • Audit Logging: All user actions are captured in the Kpow audit log.
  • Key Integrations & Deployment Features:
    • Slack Integration: Sends user actions to an operations channel.
    • Prometheus Endpoints: For integration with preferred metrics and alerting systems.
    • HTTPS Support: Easily configured with your own certificates or via a reverse-proxy.
    • Multi-Cluster Monitoring: Manage multiple Kafka clusters from a single Kpow installation.
    • Air-Gapped Environments: Well-suited due to all data being stored in local Kafka topics.
• Exposes UI at http://localhost:3000

• Zookeeper (confluentinc/cp-zookeeper:7.8.0): Coordinates the Kafka brokers.
• Kafka Brokers (confluentinc/cp-kafka:7.8.0 x3): Form the core message bus.
  • Configured with distinct internal (1909x) and external (909x) listeners for docker networking.
  • Provides basic fault tolerance with 3 nodes.
  • Accessible externally via ports 9092, 9093, 9094.

• Manages schemas (Avro, Protobuf, JSON Schema) for Kafka topics, ensuring data consistency and enabling schema evolution.
• Accessible at http://localhost:8081.
• Configured with Basic Authentication (schema_jaas.conf).
• Stores its schemas within the Kafka cluster itself.

• Framework for reliably streaming data between Apache Kafka and other systems.
• Accessible via REST API at http://localhost:8083.
• Configured to use JSON converters by default.
• Custom Connector Support: Volume mount ./resources/kpow/connector allows adding custom or third-party connectors (e.g., JDBC, S3, Iceberg - hinted by AWS_REGION).
• Manages its configuration, offsets, and status in dedicated Kafka topics.

• Build and test Kafka producers, consumers, and Kafka Streams applications against a realistic, multi-broker cluster.
• Validate application behavior during broker failures (by stopping/starting broker containers).

• Utilize Kafka Connect to ingest data into Kafka from databases, logs, or APIs using source connectors.
• Stream data out of Kafka to data lakes (S3, Iceberg), warehouses, or other systems using sink connectors.
• Test and manage connector configurations via the Connect REST API or Kpow UI.

• Define, register, and evolve schemas using Schema Registry to enforce data contracts between producers and consumers.
• Test compatibility modes and prevent breaking changes in data pipelines.

• Use Kpow to monitor cluster health, track topic/partition metrics (size, throughput), identify consumer lag, and inspect messages in real-time.
• Understand Kafka performance characteristics and troubleshoot issues within a controlled environment.

• Provides a self-contained environment to learn Kafka concepts, experiment with configurations, and explore the capabilities of the Confluent Platform and Kpow.

This stack builds a comprehensive analytics platform that erases the line between real-time stream analytics and large-scale batch processing. It achieves this by combining the power of Apache Flink, enhanced by Flex for enterprise-grade management and monitoring, with Apache Spark on a unified data lakehouse, enabling you to work with a single source of truth for all your data workloads.

This architecture is designed around a modern data lakehouse that serves both streaming and batch jobs from the same data. At its foundation, data is stored in Apache Iceberg tables on MinIO, an S3-compatible object store. This provides powerful features like ACID transactions, schema evolution, and time travel for your data.

A central Hive Metastore serves as a unified metadata catalog for the entire data ecosystem, providing essential information about the structure and location of datasets. By using a robust PostgreSQL database as its backend, the metastore reliably tracks all table schemas and metadata. This central catalog allows both Apache Flink (for low-latency streaming) and Apache Spark (for batch ETL and interactive analytics) to discover, query, and write to the same tables seamlessly, eliminating data silos.

The role of PostgreSQL is twofold: in addition to providing a durable backend for the metastore, it is configured as a high-performance transactional database ready for Change Data Capture (CDC). This design allows you to stream every INSERT, UPDATE, and DELETE from your operational data directly into the lakehouse, keeping it perfectly synchronized in near real-time.

The platform is rounded out by enterprise-grade tooling: Flex simplifies Flink management and monitoring, a Flink SQL Gateway enables interactive queries on live data streams, and a single node Spark cluster supports complex data transformations. This integrated environment is ideal for building sophisticated solutions for fraud detection, operational intelligence, and unified business analytics.

• Container: kpow from (factorhouse/flex:latest (enterprise)) or kpow-ce from (factorhouse/flex-ce:latest (community))
• Provides an enterprise-ready tooling solution to streamline and simplify Apache Flink management. It gathers Flink resource information, offering custom telemetry, insights, and a rich data-oriented UI. Key features include:
  • Comprehensive Flink Monitoring & Insights:
    • Gathers Flink resource information minute-by-minute.
    • Offers fully integrated metrics and telemetry.
    • Provides access to long-term metrics and aggregated consumption/production data, from cluster-level down to individual job-level details.
  • Simplified Management for All User Groups:
    • User-friendly interface and intuitive controls.
    • Aims to align business needs with Flink capabilities.
  • Enterprise-Grade Security & Governance:
    • Versatile Authentication: Supports DB, File, LDAP, SAML, OpenID, Okta, and Keycloak.
    • Robust Authorization: Offers Simple or fine-grained Role-Based Access Controls (RBAC).
    • Data Policies: Includes capabilities for masking and redaction of sensitive data (e.g., PII, Credit Card).
    • Audit Logging: Captures all user actions for comprehensive data governance.
    • Secure Deployments: Supports HTTPS and is designed for air-gapped environments (all data remains local).
  • Powerful Flink Enhancements:
    • Multi-tenancy: Advanced capabilities to manage Flink resources effectively with control over visibility and usage.
    • Multi-Cluster Monitoring: Manage and monitor multiple Flink clusters from a single installation.
  • Key Integrations:
    • Prometheus: Exposes endpoints for integration with preferred metrics and alerting systems.
    • Slack: Allows user actions to be sent to an operations channel in real-time.
• Exposes UI at http://localhost:3001

• JobManager (jobmanager) coordinates all tasks, handles scheduling, checkpoints, and failover. Flink UI is exposed at http://localhost:8082.
• TaskManagers (taskmanager-1, -2, -3) run user code and perform actual stream processing.
• The cluster is configured to use the central Hive Metastore for catalog services.

• Container: sql-gateway
• A REST-accessible endpoint (http://localhost:9090) for interactive Flink SQL queries against the unified catalog.

• Container: spark-iceberg
• Provides an Apache Spark environment pre-configured with Apache Iceberg support and connected to the central Hive Metastore.
• Spark Web UI for monitoring running jobs (http://localhost:4040).
• Spark History Server for reviewing completed jobs (http://localhost:18080).

• Container: hive-metastore
• A central metadata service for the entire data lakehouse. It allows both Flink and Spark to interact with the same Iceberg tables consistently.
• Uses the PostgreSQL database as its backend to durably store all metadata (schemas, partitions, table locations).
• Accessible internally at thrift://hive-metastore:9083.

• Container: postgres
• This component is the transactional and metadata backbone of the entire platform, serving two distinct and critical functions:
  1. Durable Metastore Backend: It provides the persistent storage for the Hive Metastore. All schemas, table versions, and partition information for the entire Iceberg lakehouse are stored transactionally in PostgreSQL. This makes the lakehouse catalog robust, reliable, and recoverable (Database: metastore).
  2. Transactional Workload & CDC Hub: It functions as a full-fledged relational database for application workloads. It is purpose-built for Change Data Capture (CDC), with wal_level=logical enabled by design. This configuration prepares it for seamless integration with tools like Debezium, allowing every INSERT, UPDATE, and DELETE to be captured and streamed into the Flink/Iceberg pipeline.
• Accessible at localhost:5432 (Database: fh_dev).

• MinIO provides S3-compatible object storage, acting as the data lake storage layer for Iceberg tables, Flink checkpoints, and other artifacts.
  • MinIO API at http://localhost:9000 | MinIO Console UI at http://localhost:9001 (admin/password).
• MinIO Client: A utility container that initializes MinIO by creating necessary buckets: warehouse (for Iceberg data), fh-dev-bucket, flink-checkpoints, and flink-savepoints.

• Create, manage, and query Iceberg tables using both Flink SQL for real-time writes and Spark for batch updates, all through the central Hive Metastore.
• Perform ACID-compliant operations from either engine on the same datasets.

• The architecture is purpose-built to support CDC pipelines. The wal_level=logical setting in PostgreSQL is intentionally enabled, allowing a tool like Debezium to capture every row-level change (INSERT, UPDATE, DELETE) and stream it into the data lakehouse in near real-time. This keeps the Iceberg tables continuously synchronized with the operational database.

• Use Spark to ingest data from various sources (including transactional data from PostgreSQL), perform large-scale transformations, and load it into Iceberg tables.
• Read from Iceberg tables for downstream processing, reporting, or machine learning.

• Ingest data from Kafka or CDC streams with Flink.
• Join streaming data with lookup tables in real-time.
• Write enriched, structured data directly into Iceberg tables, making it immediately available for Spark to query.

• Empower analysts to query live, streaming data via the Flink SQL Gateway and historical, large-scale data using Spark—all with familiar SQL pointed at the same tables.

This stack deploys a basic Apache Pinot cluster, a real-time distributed OLAP (Online Analytical Processing) datastore designed for ultra-low-latency analytics at scale. It includes the core Pinot components: Controller, Broker, and Server.

This architecture provides the foundation for ingesting data from batch (e.g., HDFS, S3) or streaming sources (e.g., Kafka) and making it available for analytical queries with response times often in milliseconds. Pinot is optimized for user-facing analytics, real-time dashboards, anomaly detection, and other scenarios requiring fast insights on fresh data.

Note: This configuration requires an external Apache Zookeeper instance running at zookeeper:2181 on the factorhouse network for cluster coordination, which is not defined within the Docker Compose file.

• Container: apachepinot/pinot:1.2.0
• Role: Manages the overall cluster state, handles administration tasks (like adding tables, schema management), coordinates segment assignment, and monitors node health via Zookeeper.
• Admin UI/API: Exposed externally at http://localhost:19000 (maps to internal port 9000).
• Healthcheck verifies its readiness.

• Container: apachepinot/pinot:1.2.0
• Role: Acts as the query gateway. Receives SQL queries from clients, determines which servers hold the relevant data segments, scatters the query to those servers, gathers the results, and returns the final consolidated response.
• Query Endpoint: Exposed externally at http://localhost:18099 (maps to internal port 8099).
• Depends on the Controller being healthy before starting.
• Healthcheck verifies its readiness.

• Container: apachepinot/pinot:1.2.0
• Role: Hosts data segments (shards) and executes query fragments against the data it stores. Can ingest data directly from streaming sources (Realtime Server) or load pre-built segments from deep storage (Offline Server). This configuration runs a generic Server capable of both roles depending on table setup.
• Internal API/Metrics: Exposed externally at http://localhost:18098 (maps to internal port 8098/8097 for health). Direct interaction is less common than with the Broker or Controller.
• Depends on the Broker being healthy before starting.
• Healthcheck verifies its readiness.

• All components reside on the factorhouse network.
• Relies on an external Zookeeper instance at zookeeper:2181 for coordination.
• Startup order is enforced via depends_on and healthcheck conditions: Controller -> Broker -> Server.

• Power interactive dashboards requiring millisecond query latency on potentially large, constantly updating datasets (e.g., operational monitoring, business intelligence).

• Embed analytics directly into applications where users can explore data slices and dices with immediate feedback (e.g., e-commerce site analytics, personalized recommendations).

• Query streaming event data in near real-time to identify patterns, outliers, or anomalies quickly (e.g., fraud detection, system security monitoring).

• Ingest experiment data and provide rapid aggregations and comparisons to evaluate A/B test performance.

• Provide fast, interactive querying over large volumes of log or event data for troubleshooting and analysis.

• Confluent S3 Connector: For streaming data from Kafka to S3.
• Debezium PostgreSQL Connector: For capturing row-level changes from a PostgreSQL database.
• Amazon MSK Data Generator: A tool for generating sample data for testing.
• Iceberg Kafka Connect: For sinking Kafka records into Apache Iceberg tables.

• Kafka SQL Connector: Enables Flink to read from and write to Kafka topics using SQL.
• Avro Confluent Registry: Allows Flink to work with Avro schemas stored in Confluent Schema Registry.
• Flink Faker: A connector for generating fake data streams within Flink, useful for development and testing.

• Hive SQL Connector: Allows Flink to connect to a Hive Metastore and query Hive tables.
• Supporting Libraries: Includes Hive Exec, Antlr, and Thrift libraries necessary for the Hive integration to function.

• Iceberg Flink Runtime: The core library for Flink to read from and write to Apache Iceberg tables.
• Iceberg AWS Bundle: Provides AWS-specific integrations for Iceberg, like S3 file I/O.
• Parquet SQL Formatter: Enables Flink to handle the Parquet file format.

• Hadoop Libraries: A collection of core Hadoop modules (hadoop-common, hadoop-aws, hadoop-auth) and their dependencies (aws-java-sdk-bundle, guava, etc.) required for interacting with HDFS-compatible file systems like S3.
• PostgreSQL JDBC Driver: Required for the Hive Metastore to communicate with its PostgreSQL backend database.

• Iceberg Spark Runtime: The core library for Spark to read from and write to Apache Iceberg tables.
• Iceberg AWS Bundle: Provides AWS-specific integrations for Spark, enabling it to work with Iceberg tables on S3.