Enhanced configuration validation in Redpanda

Introduction

In an earlier post, we introduced how we used Raft to improve central cluster configuration in Redpanda.

While adding this new configuration system, our engineering team also took the opportunity to improve the validation of configuration properties in order to deliver an even safer workflow for production systems. This post discusses how and why we did that.

Resolving challenges with configuration files

One of the problems with traditional configuration files is a lack of validation: your text editor cannot know whether a value you type into a file is valid or not. As a result, you may not discover an invalid value until you restart a service with the new file and something breaks. This, of course, is not ideal for your production infrastructure.

Now that we are making configuration changes via a central API rather than directly in a file, Redpanda also has the opportunity to provide immediate validation and helpful feedback to the user. New configuration is not written to the configuration store unless it passes validation.

For example, here’s what happens if we make a mistake editing log_message_timestamp_type:

Not only is the change not accepted, the message explains why, and what a valid value would be. This validation is not just done on interactive edits. Any changes using scripted rpk commands or via the Admin API are subject to the same validation.

Strong validation eliminates an entire category of operational issues that could previously happen as a result of a simple typo in a configuration file.

Beyond configuration: validating system size constraints

Configuration properties are not the only things we are validating in Redpanda 22.1. There is also brand new system sizing validation that applies to topic and partition creation.

Every partition consumes some system resources: some memory for the state of the partition and I/O buffers, file handles for the log segments on disk, and CPU time. Redpanda is efficient, but still subject to the resource limits of the server on which it runs.

In earlier versions, Redpanda would let you create larger numbers of partitions than you had system resources for. The assumption was that an expert user has their own knowledge of what their system can handle. However, this could create situations that were tough to escape: once the system had more partitions than the computer has memory, we might fail to start up, and consequently be unable to remove any partitions.

To make partition creation safer, all topic and partition creation operators are now subject to multiple checks:

• RAM per partition
• File descriptors per partition (also known as the nfile ulimit)
• Partitions per CPU core
• Available disk space

The effective limit on the number of partitions is the lowest of these. The RAM and file descriptor limits can be overridden if necessary with the topic_memory_per_partition and topic_fds_per_partition settings respectively. The defaults are to require 1MB of RAM and 10 open file handles per partition.

In future Redpanda releases, these resource thresholds will be adjusted as we further reduce the per-partition resource footprint, and validate the system with ever-higher partition counts. Look out for resource management improvements for higher partition counts in the Redpanda 22.2 release later in the year.

Under the hood: configuration property implementation details

Fellow software developers may be interested in how the validation and live-editing of configuration works in Redpanda. We use C++20 and the Seastar framework, and both of these underlying technologies help us to deliver usability improvements.

Handling live changes

In ordinary multi-threaded code, implementing notification hooks for configuration changes can be a surprisingly awkward thing to do.

• To be able to write them at runtime, configuration objects need to be protected with some kind of synchronization primitive. Taking a lock just to read configuration is expensive.
• If a configuration change requires an “on change” callback into other code, these callbacks must be written very carefully to avoid deadlocking, if they potentially access resources also in use by the thread making the configuration update.

In short: it’s a pain! This explains in part why so many projects simply give up and require a full restart to change configuration.

Seastar helps us implement live configuration changes in a robust, lightweight way:

• We store a copy of the configuration on each shard (Seastar’s term for a core-locked thread). It is always safe to read the shard-local configuration object from anywhere in our code without taking any locks.
• When applying updates to the configuration, we schedule a task on each shard to update the local copy. Because Seastar uses cooperative scheduling, there is no risk that other code will run on the same shard while we are making configuration changes. These changes, even to multiple properties, are atomic from the point of view of other code that is scheduled before or after the configuration change.
• For subsystems that need an active notification of configuration changes, they can construct a “binding” object, registered with the parent configuration object to receive callbacks on change. This also requires no extra locks or synchronization. Bindings are shard-local objects and, like their underlying properties, are safe to use from anywhere in the code.

Perhaps counterintuitively, Seastar’s performance-oriented thread-per-core programming model sometimes makes code much simpler!

Configuration property metadata

Modern C++ features enable straightforward translation of our internal types to external metadata that we can expose via our Admin API. For the Admin API, we describe properties with a Swagger-like syntax.

For example, here’s a snippet of how we derive the units parameter for a property:

template<typename T>
consteval std::string_view property_units_name() {
   using type = std::decay_t<T>;
   if constexpr (std::is_same_v<type, std::chrono::milliseconds>) {
       return "ms";
   } else if constexpr (std::is_same_v<type, std::chrono::seconds>) {
       return "s";
   …

nstexpr keyword and is_same_v give us exactly what we need for fast, highly readable code. No arcane SFINAE constructs in sight, and our underlying variables can use standard std::chrono types.

C++20 concepts are also useful for determining generic attributes of a property such as the array attribute. We can write a simple concept called is_collection that checks for a few typical methods of collections, and use this to correctly tag such properties as arrays:

template<typename T>
concept is_collection = requires(T x) {
   typename T::value_type;
   !std::is_same_v<typename T::value_type, char>;
   {x.size()};
   {x.begin()};
   {x.end()};
};

The definition of our configuration properties and their metadata benefits from C++ aggregate initialization syntax to populate the metadata and validation fields in a succinct way. For example, here’s what the constructor for a property of type bounded_property<std::optional<int>> looks like (taken from the imaginatively named configuration.cc file in the Redpanda source):

 rpc_server_tcp_recv_buf(
   *this,
   "rpc_server_tcp_recv_buf",
   "TCP receive buffer size in bytes.",
   {.example = "65536"},
   std::nullopt,
   {.min = 32_KiB, .align = 4_KiB})

In just a few lines of straightforward code, the developer can provide a description, an example for the user, and strictly enforced validation rules for the magnitude and alignment of the value.

Conclusion

Redpanda has not only improved cluster configuration and validation with the release of Version 22.1, but we have also extended validation to topic and partition sizing.

The Seastar framework enables us to deliver live configuration changes without degrading performance, and modern C++ features enable us to implement a rich configuration system with a minimum of boilerplate.

If you have specific questions about configuration validation, system sizing, or anything else, ask our engineers directly in our Slack community, or download our binary and try Redpanda for yourself.

‍