I needed a way to secure my services now that I had migrated them to Kubernetes. Previously I was using Cloudflare Access which I would highly recommend if you need to secure resources on the internet. Cloudflare made the service free while many people were working from home, which was a great idea. I didn’t feel the need to keep using it once the free period had ended given my use case was fairly small and only for my personal services. I also wanted to see if I could set up an equivalent service myself.

Enter OAuth2 Proxy. OAuth2 Proxy is

A reverse proxy and static file server that provides authentication using Providers (Google, GitHub, and others) to validate accounts by email, domain or group.

This proxy works with the Kubernetes NGINX Ingress Controller meaning I could secure my services at the Kubernetes layer rather than at the CLoudflare layer. Thankfully others have done the same thing and I followed the instructions on the External OAUTH Authentication page to get it set up with the details stepped out below.

Create an OAuth App

I used gitHub as my OAuth provider largely because that’s what the example used and it’s what I was using with Cloudflare Access. To create a new OAuth app on GitHub navigate to Settings -> Developer Settings -> OAuth Apps -> New OAuth App. It’s largely self explanatory with the Authorization callback URL pointing to the OAuth2 Proxy. I stored the client_id and client_secret in a Secret called github-oauth-dev-blog-secret which is used in the configuration below.

Create the OAuth2 Proxy instance

To set up the proxy we need a Deployment and an Ingress. I set up the Ingress to be on the same hostname as the dev instance of my blog mounted at /oauth2. I limited the proxy to only allow my GitHub username access.

# OAuth Proxy
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: devblog-oauth2-proxy
  name: devblog-oauth2-proxy
  namespace: aselford-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: devblog-oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: devblog-oauth2-proxy
    spec:
      containers:
      - name: devblog-oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:v6.0.0
        imagePullPolicy: Always
        args:
        - --provider=github
        - --upstream=file:///dev/null
        - --http-address=0.0.0.0:4180
        - --github-user="ezzizzle"  # Restrict just to my user
        env:  # Secret created externally and saved in github-oauth-dev-blog-secret
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: github-oauth-dev-blog-secret
              key: OAUTH2_PROXY_CLIENT_ID
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: github-oauth-dev-blog-secret
              key: OAUTH2_PROXY_CLIENT_SECRET
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              name: github-oauth-dev-blog-secret
              key: OAUTH2_PROXY_COOKIE_SECRET
        ports:
        - containerPort: 4180
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: devblog-oauth2-proxy
  name: devblog-oauth2-proxy
  namespace: aselford-dev
spec:
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    k8s-app: devblog-oauth2-proxy
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: devblog-oauth2-proxy
  namespace: aselford-dev
spec:
  rules:
  - host: dev-blog.aselford.dev
    http:
      paths:
      - path: /oauth2
        backend:
          serviceName: devblog-oauth2-proxy
          servicePort: 4180
  tls:
  - hosts:
    - dev-blog.aselford.dev
    secretName: dev-blog-cert

The Ingress for the blog itself needed a couple of annotations added to add the OAuth2 proxy as an auth provider.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dev-blog-ingress
  namespace: aselford-dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-production"
    # The following annotations redirect requests to the OAuth provider when not authenticated
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
spec:
  tls:
  - hosts:
    - dev-blog.aselford.dev
    secretName: dev-blog-cert
  rules:
  - host: dev-blog.aselford.dev
    http:
      paths:
      - path: /
        backend:
          serviceName: devblog
          servicePort: 80

And That’s It

That’s basically all I had to do.

For a while I could get the OAuth login to work however NGINX was returning a 503 Service Unavailable error once authenticated. This was really frustrating. Turns out during my playing around with getting things set up I had not deleted a stale ingress that pointed at a non-existant service. Deleting this fixed the issue and the full authentication flow worked.

There was just one wrinkle..

Safari 🤬

When I enabled OAuth protection for the dev instance of my blog for some reason CSS files weren’t loading.

I like minimal, not this minimal

It was limited to some local CSS files but not all. Looking in to the server logs there was a 401 for those particular CSS resources which was then attempting to redirect the user to GitHub—this was failing with a CORS error. The error was limited to Safari on macOS and iOS¹.

<!-- Working -->
<link rel="stylesheet" href="/css/images.css">
<link rel="stylesheet" href="/css/article.css">
<!-- Not Working -->
<link rel="stylesheet" href="/css/coder.min.32..Qs=" crossorigin="anonymous" media="screen">
<link rel="stylesheet" href="/css/coder-dark.min.e7...PY=" crossorigin="anonymous" media="screen">

The CSS files that weren’t working had a crossorigin="anonymous" tag. According to MDN:

The “anonymous” keyword means that there will be no exchange of user credentials via cookies, client-side SSL certificates or HTTP authentication as described in the Terminology section of the CORS specification, unless it is in the same origin.

Unfortunately for me, there’s a bug in Safari where the crossorigin="anonymous" setting refuses to send credentials even if it’s from the same origin… 😞 This bug was reported in 2017 and only fixed in May this year; it’s fixed in the Safari Technology Preview but not yet in the main version of Safari.

Sadly I will have to wait until that fix makes it in to production before I can access the dev version of my blog from my main browser.