Kafka authentication

Apache Kafka® is a distributed streaming platform extensively used to construct real-time data pipelines. It’s often used to transmit sensitive data between systems, making security an essential concern. Kafka offers customizable security features, like authentication, authorization, encryption, and auditing, to ensure the safety of your data and applications. Authentication plays a critical role in securing Kafka by verifying user and system identities.

This chapter focuses on Kafka authentication mechanisms. We explain different security protocols, how to configure them, and some best practices.

Summary of key Kafka authentication concepts

Authentication basics

Authentication is the first layer of protection that secures your Kafka cluster. A Kafka broker verifies the identity of clients(producers or consumers) and other brokers to ensure that only authorized entities access its resources. Kafka clients must prove their identity before producing or consuming messages from topics. Similarly, brokers authenticate each other to ensure secure data exchange between nodes in the cluster.

KafkaPrincipal

KafkaPrincipal represents the identity of the user or service that interacts with the Kafka cluster. It helps Kafka identify that the service or user is what it claims to be.

For example, clients present their KafkaPrincipal when attempting to connect. Kafka verifies their identity and determines whether they can make the request. No entity can access the restricted Kafka resources without a valid KafkaPrincipal.

Kafka associates the KafkaPrincipal with requests even when authentication is not enabled; in that case, the Principal associated is ANONYMOUS.

Listeners

Listeners are network endpoints that Kafka brokers use to accept incoming connections from clients and other brokers. Based on requirements, you can configure each listener to use different security protocols (explained later) to establish secure communication channels.

Listeners are specified in the server.properties file, where you can define multiple listeners for different security protocols. Below is an example configuration snippet:

# Define listeners for different network interfaces and security protocols
listeners=SASL_SSL://:9092,SSL://10.0.0.2:9093,SSL://10.0.0.2:9094

Advertised listeners are the addresses that Kafka brokers advertise to clients and other brokers for connection purposes. In environments with complex network setups (e.g., Kubernetes, cloud environments, or when dealing with NATs), the Kafka broker might have different internal and external addresses. Advertised listeners ensure that clients receive the correct address for the network they are on.

You can specify the listeners in the server.properties file. Below is an example configuration for advertised listeners.

# Define advertised listeners to be exposed to clients and other brokers
advertised.listeners=SASL_SSL://broker1.example.com:9092,SSL://internal.broker1.example.com:9093,SSL://broker1.local:9094

Security protocols in Kafka authentication

You can configure different security protocols for authentication.

SSL/TLS

The SSL/TLS protocol requires client authentication through mutual TLS exchange. Clients must present a valid SSL certificate to connect. You can use Java’s keytool utility or a tool like OpenSSL to create the certificates.

The keystore file stores private keys and the corresponding public key certificates used to identify the server or client. The truststore file stores public key certificates of trusted Certificate Authorities (CAs).

You can configure listeners, keystore, and truststore inside the server.properties file for brokers, as shown.

# Accept SSL-encrypted connections from clients and other brokers
listeners=SSL://:9092
# Configure Keystore and Truststore for brokers
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=keystore_password
ssl.key.password=key_password
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=truststore_password
# If client authentication is required
ssl.client.auth=required

You also have to provide the security protocol details to the clients. These configurations can be done in a client.properties file or directly within your producer/consumer code. Examples for both cases are below:

Separate client.properties file.

# Security protocol setting for clients connecting to Kafka brokers
security.protocol=SSL
# Truststore configuration for clients
ssl.truststore.location=/path/to/client.truststore.jks
ssl.truststore.password=truststore_password
# If client authentication is required, specify the keystore details
ssl.keystore.location=/path/to/client.keystore.jks
ssl.keystore.password=your-keystore-password
ssl.key.password=your-key-password

Our Apache Kafka tutorial contains examples of Kafka consumer and producer code. You can add the below configurations to the properties object in the Producer or Consumer class.

// SSL Configuration
props.put("security.protocol", "SSL");
props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, "/path/to/client.truststore.jks");
props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, "your-truststore-password");
// If client authentication is required, specify the keystore details
props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, "/path/to/client.keystore.jks");
props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, "your-keystore-password");
props.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, "your-key-password");

SASL_SSL

Simple Authentication and Security Layer over SSL (SASL_SSL) combines the encryption provided by SSL with the authentication mechanisms offered by SASL (Simple Authentication and Security Layer). SASL is not a protocol but a framework that provides data security services like authentication via replaceable mechanisms in other connection-oriented protocols. Kafka supports several SASL mechanisms:

• GSSAPI (Kerberos) for environments with Kerberos authentication, such as Active Directory or OpenLDAP.
• PLAIN for simple username/password authentication.
• SCRAM with SHA-256 or SHA-512 for secure username/password authentication.
• OAUTHBEARER uses OAuth 2.0 Bearer tokens for authentication.

You must configure listeners, SASL mechanism, keystore, and truststore inside the server.properties file.

# Accept SASL_SSL-encrypted connections from clients and other brokers
listeners=SASL_SSL://:9092
# Specify the SASL mechanism
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
sasl.enabled.mechanisms=SCRAM-SHA-512
# Configure Keystore and Truststore for brokers
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=keystore_password
ssl.key.password=key_password
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=truststore_password
# Client authentication is required for SASL_SSL
ssl.client.auth=required

You also configure clients via a properties file or application code.

Separate client.properties file:

# Security protocol setting for clients connecting to Kafka brokers
security.protocol=SASL_SSL
# SASL mechanism configuration
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="<your-username>" \
  password="<your-password>";
# Truststore configuration for clients
ssl.truststore.location=/path/to/client.truststore.jks
ssl.truststore.password=truststore_password
# Keystore configuration for client authentication
ssl.keystore.location=/path/to/client.keystore.jks
ssl.keystore.password=your-keystore-password
ssl.key.password=your-key-password

Alternatively, add the configurations below to the properties object in the Producer or Consumer class.

// Set the Security Protocol to SASL_SSL
props.put("security.protocol", "SASL_SSL");
# SASL mechanism configuration
props.put(SaslConfigs.SASL_MECHANISM, "SCRAM-SHA-512");
props.put(SaslConfigs.SASL_JAAS_CONFIG, "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"your-username\" password=\"your-password\";");
# Truststore configuration for clients
props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, "/path/to/client.truststore.jks");
props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, "your-truststore-password");
# Keystore configuration for client authentication
props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, "/path/to/client.keystore.jks");
props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, "your-keystore-password");
props.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, "your-key-password");

Recommendations

You may inadvertently expose sensitive information in your configuration files or code. Use secure vaults like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault to securely store and access secrets. You can also use environment variables to keep sensitive information out of source code and configuration files. Set environment variables at the system or container level where your Kafka clients run. Your application reads these variables at runtime to configure connections to the Kafka cluster.

Other general recommendations to strengthen Kafka authentication include:

• Use complex passwords and enforce regular password rotations.
• Enable comprehensive auditing and logging features to track failed authentication events and other errors.
• Keep SSL certificates up-to-date.

You can define an SSL certificate renewal task in your CI/CD pipeline to prevent vulnerabilities caused by expired or compromised certificates.

Last thoughts

Kafka is a complex system with complex configurations, and securing it becomes harder as the system scales. You must maintain security configurations and access scope information for the wide range of users and applications accessing your Kafka clusters.

Redpanda is a drop-in Kafka replacement engineered to address Kafka's inherent limitations. It supports both mTLS principal extraction and SASL authentication methods with multiple listeners. You can support various Kafka clients using authentication schemes to connect to your cluster.

Redpanda offers a fully managed service called Redpanda Cloud to simplify cluster management even further. You can choose to host clusters in your own cloud with provisioning, maintenance, and monitoring fully managed by Redpanda.

‍