3 Steps to Apache Zookeeper Authentication
Coordinating and managing a service in a distributed environment can get complicated, as it requires you to develop and maintain high-quality software that is free from security threats. Thankfully, there are numerous tools available that make this task a little less demanding.
When we boil our choices down to open source, the most popular application to help us achieve our goal is Apache ZooKeeper, a centralized service for maintaining configuration information, providing distributed synchronization, and group services.
ZooKeeper is widely used by other open source tools (for example Kafka), and integrated into various Hadoop distribution platforms.
Adding Authentication Support to Apache ZooKeeper
In this post, we will focus on security concerns in ZooKeeper and present how to add authentication support.
First versions of the tool did not support authentication or authorization of servers participating in the leader election and the quorum forming process. This could result in a leakage of sensitive data and errors in the system operation.
Fortunately, the recent versions of ZooKeeper allow suitable configuration of instances. We will present such configuration in the three simple steps below.
To showcase our example, we won’t delve into more advanced concepts such as quorum and leader election. Instead, we will describe those aspects of the configuration process that ensure high-level data security. Quorum should authenticate and authorize requests coming from other instances.
Prerequisites to Hardening ZooKeeper Security – Kerberos Instance
Hardening ZooKeeper security is a part of the configuration that needs to be applied before production deployment.
Starting from version 3.4.10+, Zookeeper supports mutual server-to-server authentication using SASL, which provides a layer around Kerberos authentication. Server-to-server authentication between instances reduces the risk of data falsification in an unsecured network.
Configuring ZooKeeper to use Kerberos for client-to-server or server-to-server authentication requires an MIT Kerberos or Microsoft Active Directory instance to be up, running, and reachable. Therefore, before moving forward with the security configuration, make sure you have installed Kerberos.
1. ZooKeeper Security – Server-to-Server Authentication
ZooKeeper stores a list of servers in a cluster in the zoo.cfg file. These servers are authorized and authenticated by comparing a server’s FQDN (fully.qualified.domain.name) extracted from the service principal name.
To configure the server-to-server authentication, follow the steps below:
- Create a service principal for the ZooKeeper server using a fully-qualified domain name (FQDN):
kadmin: addprinc -randkey zookeeper/fqdn.example.com@YOUR-REALM
- Create a keytab file for the ZooKeeper server:
kadmin: xst -k zookeeper.keytab zookeeper/fqdn.example.com@YOUR-REALM
- Copy the keytab file to the ZooKeeper configuration directory.
- Set up the Java Authentication and Authorization Service (JAAS) by creating a jaas.conf file in the configuration directory.
- Add the following setting to the env file located in the configuration directory
export JVMFLAGS=”-Djava.security.auth.login.config=/etc/zookeeper/conf/jaas.conf”
- Add the following settings to the cfg configuration file:
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.auth.kerberos.servicePrincipal=servicename/_HOST
quorum.cnxn.threads.size=20
The above configuration presents a very simple use case. For a more complex example, please refer, e.g., to Github.
2. ZooKeeper Security – Client-Server Authentication
In addition to the server-to-server authentication, you should also configure client-to-server mutual verification. ZooKeeper supports pluggable authentication schemes. It uses Access Control Lists (ACLs) to control access to its znodes (data nodes of a ZooKeeper data tree).
The ACL implementation is quite similar to UNIX file access permissions. A node may have any number of <scheme:expression,perms> pairs. The left member of the pair specifies the authentication scheme, while the right member indicates permissions (ACL pertains only to a specific znode).
ZooKeeper supports the following permissions:
- CREATE: create a child node
- READ: get data from a node and list its children
- WRITE: set data for a node
- DELETE: delete a child node
- ADMIN: set permissions
And it has the following built-in schemes:
- world – anyone
- sasl – for kerberos/sasl authentication
- digest – for MD5 hash
- ip – IP used as ACL ID identity
3. Verifying ZooKeeper Settings
After enabling Kerberos authentication and restarting the Zookeeper cluster, you can verify settings by following these steps:
- Start the ZooKeeper client:
zookeeper-client -server fqdn.example.com:port
- Use the CLI to create a protected znode:
create /znode1 znode1data sasl:zkcli@{{YOUR-REALM}}:cdwra
- Verify if the ACL is set correctly:
getAcl /znode1
ZooKeeper Level Hard. Are You Ready?
We have covered the essential aspects of hardening ZooKeeper security, tuning the default behavior, and enabling the built-in features. Now the real challenge begins.
The next step is to dive deep into the available documentation and try out different settings and configurations on your own. Setting up more advanced features can be maddening because of the lack of sufficient support and frequent changes between releases. However, the outcome will probably compensate you for the effort and provide a sense of heightened security.
Share Your Thoughts!
If you come across a particular configuration or find some handy tips, please share them in the comments section, so that we can all benefit. Let us also know if you spot any errors in the examples or think that we missed anything. All feedback is welcome!
What’s your opinion on ZooKeeper’s authentication features? Share your thoughts in the comments below and stay tuned for the next posts!
What Can We Do For Your Business?