In mid-August 2016, fromAtoB switched from running on a few hand-managed bare-metal servers to Google Cloud Platform (GCP), using saltstack, packer, and terraform to
programmatically define and manage our infrastructure. After this migration, it
was relatively straightforward to setup and expose our internal services such as kibana,
grafana, and prometheus to the internet at large with a small set of salt states
lego on individual machines running
the services managed by systemd.
Then, mid-September, we migrated our Ruby on Rails code to run within Kubernetes
on Google Container Engine (GKE). This was a big change for us on almost all parts of
the stack, as deployments no longer worked with
Capistrano, but via
and our CI setup had to change dramatically to allow for docker images to be
built and pushed during CI. However, everything went rather smoothly in the end.
Next, we wanted to migrate our internal services to run within Kubernetes, too -
but we did not have an easy solution to managing Ingress. The Rails application
ran as a
NodePort service connected to a terraform managed GCP HTTP Load Balancer,
which had all of our main site SSL certificates. Marrying this setup to Let’s Encrypt
would not have been very easy, as the GCP HTTP Load Balancer does not support TLS-SNI
(at the time of this article).
TLS-SNI and Google Cloud Platform Woes
On GCP, the HTTP load balancers do not support TLS-SNI, which means you need a
new frontend IP address per SSL certificate you have. For internal services, this
is a pain, as you cannot point a wildcard DNS entry to a single IP, like
and then have everything just work. Luckily, we realized that using a TCP
load balancer with the Nginx IngressController
would work just as well, and support TLS-SNI no problem.
Setting this up was straightforward, by creating a Kubernetes
DaemonSet that runs
the IngressController on every node, and then pointing a TCP Load Balancer+HealthCheck
to each GKE instance we run.
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nginx namespace: nginx-ingress spec: template: metadata: labels: app: nginx spec: terminationGracePeriodSeconds: 60 containers: - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3 name: nginx imagePullPolicy: Always env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace readinessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP livenessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 80 hostPort: 80 - containerPort: 443 hostPort: 443 args: - /nginx-ingress-controller - --default-backend-service=nginx-ingress/default-http-backend - --nginx-configmap=nginx-ingress/nginx-ingress-controller
Adding Oauth 2 Authentication
It’s relatively important to expose your internal dashboards and services to the outside world with authentication, and oauth2 proxy makes this super simple. We like to run it inside the same Pod that manages our service deployment - for Kibana this means our deployment looks like
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kibana namespace: default spec: replicas: 1 revisionHistoryLimit: 2 template: metadata: labels: app: kibana spec: containers: - image: kibana:5.0.1 imagePullPolicy: Always name: kibana env: - name: ELASTICSEARCH_URL value: "http://elasticsearch:9200" resources: limits: cpu: 200m memory: 200Mi requests: cpu: 50m memory: 100Mi ports: - containerPort: 5601 - name: oauth2-proxy image: a5huynh/oauth2_proxy args: - "-upstream=http://localhost:5601/" - "-provider=github" - "-cookie-secure=true" - "-cookie-expire=168h0m" - "-cookie-refresh=60m" - "-cookie-secret=SECRET COOKIE" - "-cookie-domain=kibana.fromatob.com" - "-http-address=0.0.0.0:4180" - "-redirect-url=https://kibana.fromatob.com/oauth2/callback" - "-github-org=fromAtoB" - "-email-domain=*" - "-client-id=github oauth ID" - "-client-secret=github oauth secret" ports: - containerPort: 4180
and the service for the deployment is just as straightforward -
apiVersion: v1 kind: Service metadata: name: kibana namespace: default spec: ports: - port: 80 targetPort: 4180 protocol: TCP selector: app: kibana
Adding Let’s Encrypt
Fortunately for us, integrating Let’s Encrypt with Kubernetes via the Nginx Ingress
Controller is easy, thanks to the fantastic kube-lego which automatically provisions
SSL certificates for Kubernetes
Ingress Resources with the addition of a few
After setting up the appropriate service and deployment for Kibana, simply
Ingress resource results in the Nginx being set up and a Let’s Encrypt
certificate provisioned for the domain.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kibana namespace: default annotations: kubernetes.io/tls-acme: "true" kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - kibana.fromatob.com secretName: kibana-tls rules: - host: kibana.fromatob.com http: paths: - path: / backend: serviceName: kibana servicePort: 80
With the right setup, it’s super easy to expose protected, HTTPS resources from Kubernetes, if you just want to copy our setup, the manifests we use in production are available on GitHub.
Going forward, we want to investigate setting up mate to
automatically provision DNS records from the very same
Ingress resources that manage everything else, and switch our main production site to use the same style of
ingress as everything else within Kubernetes.