KSQL
KSQL is an open-source streaming SQL engine that implements continuous, interactive queries.
Use KSQL to query, read, write, and process data in real-time, at scale, through SQL commands. KSQL interacts directly with the Kafka Streams API, eliminating the need for a Java application.
The KSQL data flow architecture is designed such that the user interacts with the KSQL server and the KSQL server interacts with the HPE Ezmeral Data Fabric Streams server.
Use Cases
Common use cases include fraud detection, personalization, notifications, real-time analytics, and sensor data and IoT.
Architecture
A set of KSQL processes run as a cluster, and the KSQL server process completes queries. You can dynamically add more processing capacity by starting more instances of the KSQL server. These instances are fault-tolerant: if one fails, the others continue the work. Queries are launched using the interactive KSQL command line client, which sends commands to the cluster over a REST API. The command line allows you to inspect the available streams and tables, issue new queries, check the status of and terminate running queries.
- KSQL Server
- The KSQL server runs the engine that completes KSQL queries. This includes processing, reading, and writing data to and from the target Kafka cluster. KSQL servers form KSQL clusters and can run in containers, virtual machines, and bare-metal machines. You can add and remove servers in a KSQL cluster during live operations to elastically scale processing capacity. You can deploy different KSQL clusters to achieve workload isolation.
- KSQL CLI
- You can interactively write KSQL queries through the KSQL command line interface (CLI). The KSQL CLI acts as a client to the KSQL server.
KSQL Deployment Modes
You can deploy KSQL queries through Interactive or Non-Interactive mode.
- Interactive Mode
-
In Interactive mode, users interact with the KSQL server through a REST API, such as the KSQL CLI. Interactive mode is useful when users need to write and verify their queries interactively on a shared KSQL cluster. In interactive KSQL clusters, the authenticated KSQL user must have open access to create, read, write, delete topics, and use of any consumer group.
Starting in EEP 7.0.0 (Core 6.2.0 and KSQL 5.2.1.0), Interactive mode is recommended for production.
- Non-Interactive Mode (Headless Mode)
-
Non-interactive mode supports locked-down or “headless” deployment scenarios where interactive use of the KSQL cluster is disabled, thereby preventing KSQL CLI access. In this mode, you can write queries to an SQL file, which allows for version control, and lock down access to KSQL servers to prevent users from interacting directly with the KSQL cluster.
KSQL Security
By default, KSQL is secured when installed on secured clusters (clusters secured by the data fabric ticket-based security). See KSQL Security for additional information.