Let's Encrypt, OAuth 2, and Kubernetes Ingress

In mid-August 2016, fromAtoB switched from running on a few hand-managed bare-metal servers to Google Cloud Platform (GCP), using saltstack, packer, and terraform to programmatically define and manage our infrastructure. After this migration, it was relatively straightforward to setup and expose our internal services such as kibana, grafana, and prometheus to the internet at large with a small set of salt states that managed oauth2_proxy, nginx, and lego on individual machines running the services managed by systemd.

Then, mid-September, we migrated our Ruby on Rails code to run within Kubernetes on Google Container Engine (GKE). This was a big change for us on almost all parts of the stack, as deployments no longer worked with Capistrano, but via kubectl, and our CI setup had to change dramatically to allow for docker images to be built and pushed during CI. However, everything went rather smoothly in the end.

Next, we wanted to migrate our internal services to run within Kubernetes, too - but we did not have an easy solution to managing Ingress. The Rails application ran as a NodePort service connected to a terraform managed GCP HTTP Load Balancer, which had all of our main site SSL certificates. Marrying this setup to Let’s Encrypt would not have been very easy, as the GCP HTTP Load Balancer does not support TLS-SNI (at the time of this article).

TLS-SNI and Google Cloud Platform Woes

On GCP, the HTTP load balancers do not support TLS-SNI, which means you need a new frontend IP address per SSL certificate you have. For internal services, this is a pain, as you cannot point a wildcard DNS entry to a single IP, like *.fromatob.com and then have everything just work. Luckily, we realized that using a TCP load balancer with the Nginx IngressController would work just as well, and support TLS-SNI no problem.

Setting this up was straightforward, by creating a Kubernetes DaemonSet that runs the IngressController on every node, and then pointing a TCP Load Balancer+HealthCheck to each GKE instance we run.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx
  namespace: nginx-ingress
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3
        name: nginx
        imagePullPolicy: Always
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        args:
        - /nginx-ingress-controller
        - --default-backend-service=nginx-ingress/default-http-backend
        - --nginx-configmap=nginx-ingress/nginx-ingress-controller

Adding Oauth 2 Authentication

It’s relatively important to expose your internal dashboards and services to the outside world with authentication, and oauth2 proxy makes this super simple. We like to run it inside the same Pod that manages our service deployment - for Kibana this means our deployment looks like

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: default
spec:
  replicas: 1
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - image: kibana:5.0.1
        imagePullPolicy: Always
        name: kibana
        env:
        - name: ELASTICSEARCH_URL
          value: "http://elasticsearch:9200"
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 50m
            memory: 100Mi
        ports:
        - containerPort: 5601
      - name: oauth2-proxy
        image: a5huynh/oauth2_proxy
        args:
          - "-upstream=http://localhost:5601/"
          - "-provider=github"
          - "-cookie-secure=true"
          - "-cookie-expire=168h0m"
          - "-cookie-refresh=60m"
          - "-cookie-secret=SECRET COOKIE"
          - "-cookie-domain=kibana.fromatob.com"
          - "-http-address=0.0.0.0:4180"
          - "-redirect-url=https://kibana.fromatob.com/oauth2/callback"
          - "-github-org=fromAtoB"
          - "-email-domain=*"
          - "-client-id=github oauth ID"
          - "-client-secret=github oauth secret"
        ports:
        - containerPort: 4180

and the service for the deployment is just as straightforward -

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
spec:
  ports:
  - port: 80
    targetPort: 4180
    protocol: TCP
  selector:
    app: kibana

Adding Let’s Encrypt

Fortunately for us, integrating Let’s Encrypt with Kubernetes via the Nginx Ingress Controller is easy, thanks to the fantastic kube-lego which automatically provisions SSL certificates for Kubernetes Ingress Resources with the addition of a few simple annotations.

After setting up the appropriate service and deployment for Kibana, simply creating an Ingress resource results in the Nginx being set up and a Let’s Encrypt certificate provisioned for the domain.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  namespace: default
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - kibana.fromatob.com
    secretName: kibana-tls
  rules:
  - host: kibana.fromatob.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 80

Conclusion

With the right setup, it’s super easy to expose protected, HTTPS resources from Kubernetes, if you just want to copy our setup, the manifests we use in production are available on GitHub.

Going forward, we want to investigate setting up mate to automatically provision DNS records from the very same Ingress resources that manage everything else, and switch our main production site to use the same style of ingress as everything else within Kubernetes.


Want to work with Ian? Say Hi!

Join the team

View open positions