Three-headed dog meet cockroach, part 6: CockroachDB, MIT Kerberos, HAProxy and Docker Compose

Today, I'm going to try to simplify our architecture or at least management of Kerberos artifacts as they relate to CockroachDB by introducing a load balancer. Given a presence of LB, we can obfuscate the CockroachDB cluster architecture from Kerberos and ease the managment of Kerberos keytabs as well as service principal names.

CockroachDB with MIT Kerberos, HAProxy and Docker Compose - Simplified

Articles covering CockroachDB and Kerberos

I find the topic of Kerberos very interesting and my colleagues commonly refer to me for help with this complex topic. I am by no means an expert at Kerberos, I am however familiar enough with it to be dangerous. That said, I've written multiple articles on the topic which you may find below:

In my current role, I find new opportunities to learn and grow as an engineer every day. This is the reason I gravitate towards client-facing roles. Having access to many customers and their use cases, broadens my understanding and expertise. This series of articles is pushing me to understand security concepts further. Today, we're going to focus on clearing one misconception that I may have led my readers on. If you recall my original three articles on CockroachDB and Kerberos, I walked through the steps of creating SPN and keytab entries for every node in the cluster. I was recently conducting an internal enablement on CockroachDB and Kerberos and I was faced with a question from a colleague that forced further research into the topic. Turns out it is not necessary to create an SPN and an associated keytab for each node. You can get by with a single SPN and keytab entry if you're using a load balancer. Load balancer will authenticate with Kerberos KDC and with the client and that should be enough for the purposes of this conversation. In all the previous articles, I chose not to use a load balancer to keep the architecture simple. Today, I'm going to fill the gaps in our understanding and hopefully close the book on this concept.

  1. Copy the cockroach-gssapi directory to your machine
git clone https://github.com/dbist/cockroach-docker
cd cockroach-docker/cockroach-gssapi
  1. Run ./up.sh script to start the environment
./up.sh
cockroach uses an image, skipping
Building roach-cert
Step 1/15 : FROM cockroachdb/cockroach:v20.1.3 AS generator
 ---> 25bee4f016c4
...
SET CLUSTER SETTING

Time: 8.1053ms
  1. Check to make sure all containers are up
docker-compose ps
cockroach    /cockroach/cockroach.sh st ...   Up      26257/tcp, 8080/tcp
kdc          /start.sh                        Up
lb           /docker-entrypoint.sh hapr ...   Up      0.0.0.0:26257->26257/tcp,
                                                      0.0.0.0:8080->8080/tcp,
                                                      0.0.0.0:8081->8081/tcp
psql         /start.sh                        Up      5432/tcp
roach-cert   /bin/sh -c tail -f /dev/null     Up

We want to make sure we have a load balancer container served by haproxy.

  1. Connect to the psql container and pass the connection string.

We're going to connect to the loadbalancer lb instead of cockroach node in our connection string.

psql "postgresql://lb:26257/defaultdb?sslmode=verify-full&sslrootcert=/certs/ca.crt" -U tester
psql (9.5.22, server 9.5.0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 128, compression: off)
Type "help" for help.

defaultdb=> \q

There are a few things at play here. First, we added postgres/lb@EXAMPLE.COM as our SPN to our KDC and also populated keytab with that information.

So if you look in kdc/Dockerfile, you should see:

RUN kdb5_util create -s -P kpass \
  && kadmin.local -q "addprinc -pw psql tester@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey postgres/cockroach@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey postgres/lb@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey customspn/cockroach@EXAMPLE.COM" \
  && kadmin.local -q "addprinc -randkey customspn/lb@EXAMPLE.COM"

I kept postgres/cockroach@EXAMPLE.COM just so I could connect to cockroach node directly but it is no longer necessary. The only SPN and keytab we need is spn/lb@REALM. The start.sh file contains the following:

kadmin.local -q "ktadd -k /keytab/crdb.keytab postgres/lb@EXAMPLE.COM"

# This is an example of overriding postgres SPN
kadmin.local -q "ktadd -k /keytab/crdb.keytab customspn/lb@EXAMPLE.COM"

I'm including a customspn just for consistency. So in summary, the only things you need to do are the following:

kadmin.local -q "addprinc -randkey postgres/lb@EXAMPLE.COM
kadmin.local -q "ktadd -k /keytab/crdb.keytab postgres/lb@EXAMPLE.COM

and that's that, now whether you add new nodes or remove old, as long as your load balancer is up, you should have a single SPN and keytab entry to manage. Obviously if you have multiple load balancers, which we highly recommend, you should have those LBs in the kdc database and have approapriate keytabs available. It is hard to emphacise the simplicity of this in a small environment but when you consider hundreds of nodes and just a few load balancer instances, this starts to look valuable.

Finally, let's talk about the caveats, in my roach-cert Dockerfile, I had to add lb to the node cert and you have to do that for each node, as when you connect via the load balancer, the node's cert must have lb hostname as part of SAN.

openssl x509 -in /certs/node.crt -text | grep "X509v3 Subject Alternative Name" -A 1
            X509v3 Subject Alternative Name:
                DNS:cockroach, DNS:lb

And if you have multiple load balancers, they all need to be added to the node certs. Also, in my docker compose file, you have to populate the psql hosts file with the IP of the load balancer node, not necessarily the cockroach node.

    extra_hosts:
      - "cockroach:172.28.1.4"
      - "lb:172.28.1.5"

I'm including cockroach anyway but it is not necessary if you only use postgres://lb:26257 as your connection string, having cockroach there will also allow to connect with the node's name, not just load balancer.

Hope this clarifies things for you as it did for me. If you have any feedback, please don't hesitate to reach out.

view raw cockroachdb-kerberos-simplified.md hosted with ❤ by GitHub

Comments

Post a Comment

Popular posts from this blog

Vista Vulnerability Report mentions Ubuntu 6.06 LTS

Update: I posted this on MIXX since DIGG was taken care of by someone already, except that that user didn't really write a good preview for it so it got buried. This report is generating some buzz though. I emailed this report to Mark Shuttleworth of Ubuntu fame and little did I know, HE ANSWERED!!!! Whoever answered me on that side of the e-mail pipe liked the report and was wondering if it was official and whether he could quote from it. It definitely is official and I forwarded him the e-mail with this report, he said he will pass it down internally. That made my day, it's amazing that a man of his statute, that's if it was him, answers e-mails from regular people like me. That's what I call a real genuine pioneer! Thank you Mark and all of the Ubuntu Staff. Read
Read more

Running CockroachDB with Docker Compose and Minio, Part 2

This is my second post on creating a multi-service architecture with docker-compose. This is meant to be a learning exercise and typically docker-compose is used to set up a local development environment rather than a production-ready set up. I regularly, find myself building these environments to reproduce customer bugs. For a production-specific application, refer to your platform vendor documentation. At some later time, I will cover Kubernetes deployments that can be used as a stepping stone for a real-world application. Until then, let's focus on the task at hand. We're building a microservice architecture with CockroachDB writing changes in real-time to an S3 bucket in JSON format. S3 bucket is served by a service called Minio. It can act like an S3 appliance on premise or serve as a local gateway to your cloud storage. Let's dig in:
Read more

VirtualBox options to start VM in Normal, Detached and Headless Modes

Image
I've been an avid user of VirtualBox for a long time! Much of what I've learned in my career had something to do with VirtualBox. I do all my testing and new tutorials in VirtualBox, I use Hortonworks Sandbox VirtualBox image for my work. This is probably not a new feature but it is definitely new to me because I just noticed it, VirtualBox has an option to run in different modes, at least from the GUI perspective. I know there's a lot of untapped potential in using the VBox CLI but I haven't had much need for it yet. Back to the topic, you can now launch your VMs in headless and detached modes right from the GUI! Usually, when I launch a VM in normal way, the VM window just hangs there and is pretty annoying. I prefer to SSH to a running VM from a console instead of using the VM window. Now I can launch a VM in headless mode and it's business as usual. No extra windows, and I bet the memory toll on my machine is also smaller. When you click on start your VM, you no...
Read more