I needed a way to secure my services now that I had migrated them to Kubernetes. Previously I was using Cloudflare Access which I would highly recommend if you need to secure resources on the internet. Cloudflare made the service free while many people were working from home, which was a great idea. I didn’t feel the need to keep using it once the free period had ended given my use case was fairly small and only for my personal services. I also wanted to see if I could set up an equivalent service myself.
Enter OAuth2 Proxy. OAuth2 Proxy is
A reverse proxy and static file server that provides authentication using Providers (Google, GitHub, and others) to validate accounts by email, domain or group.
This proxy works with the Kubernetes NGINX Ingress Controller meaning I could secure my services at the Kubernetes layer rather than at the CLoudflare layer. Thankfully others have done the same thing and I followed the instructions on the External OAUTH Authentication page to get it set up with the details stepped out below.
Create an OAuth App
I used gitHub as my OAuth provider largely because that’s what the example used and it’s what I was using with Cloudflare Access. To create a new OAuth app on GitHub navigate to Settings -> Developer Settings -> OAuth Apps -> New OAuth App. It’s largely self explanatory with the
Authorization callback URL pointing to the OAuth2 Proxy. I stored the
client_secret in a Secret called
github-oauth-dev-blog-secret which is used in the configuration below.
Create the OAuth2 Proxy instance
To set up the proxy we need a Deployment and an Ingress. I set up the Ingress to be on the same hostname as the dev instance of my blog mounted at
/oauth2. I limited the proxy to only allow my GitHub username access.
# OAuth Proxy apiVersion: apps/v1 kind: Deployment metadata: labels: k8s-app: devblog-oauth2-proxy name: devblog-oauth2-proxy namespace: aselford-dev spec: replicas: 1 selector: matchLabels: k8s-app: devblog-oauth2-proxy template: metadata: labels: k8s-app: devblog-oauth2-proxy spec: containers: - name: devblog-oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:v6.0.0 imagePullPolicy: Always args: - --provider=github - --upstream=file:///dev/null - --http-address=0.0.0.0:4180 - --github-user="ezzizzle" # Restrict just to my user env: # Secret created externally and saved in github-oauth-dev-blog-secret - name: OAUTH2_PROXY_CLIENT_ID valueFrom: secretKeyRef: name: github-oauth-dev-blog-secret key: OAUTH2_PROXY_CLIENT_ID - name: OAUTH2_PROXY_CLIENT_SECRET valueFrom: secretKeyRef: name: github-oauth-dev-blog-secret key: OAUTH2_PROXY_CLIENT_SECRET - name: OAUTH2_PROXY_COOKIE_SECRET valueFrom: secretKeyRef: name: github-oauth-dev-blog-secret key: OAUTH2_PROXY_COOKIE_SECRET ports: - containerPort: 4180 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: k8s-app: devblog-oauth2-proxy name: devblog-oauth2-proxy namespace: aselford-dev spec: ports: - name: http port: 4180 protocol: TCP targetPort: 4180 selector: k8s-app: devblog-oauth2-proxy --- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: devblog-oauth2-proxy namespace: aselford-dev spec: rules: - host: dev-blog.aselford.dev http: paths: - path: /oauth2 backend: serviceName: devblog-oauth2-proxy servicePort: 4180 tls: - hosts: - dev-blog.aselford.dev secretName: dev-blog-cert
The Ingress for the blog itself needed a couple of annotations added to add the OAuth2 proxy as an auth provider.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: dev-blog-ingress namespace: aselford-dev annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/ssl-redirect: "true" cert-manager.io/cluster-issuer: "letsencrypt-production" # The following annotations redirect requests to the OAuth provider when not authenticated nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth" nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri" spec: tls: - hosts: - dev-blog.aselford.dev secretName: dev-blog-cert rules: - host: dev-blog.aselford.dev http: paths: - path: / backend: serviceName: devblog servicePort: 80
And That’s It
That’s basically all I had to do.
For a while I could get the OAuth login to work however NGINX was returning a 503 Service Unavailable error once authenticated. This was really frustrating. Turns out during my playing around with getting things set up I had not deleted a stale ingress that pointed at a non-existant service. Deleting this fixed the issue and the full authentication flow worked.
There was just one wrinkle..
When I enabled OAuth protection for the dev instance of my blog for some reason CSS files weren’t loading.
I like minimal, not this minimal
It was limited to some local CSS files but not all. Looking in to the server logs there was a 401 for those particular CSS resources which was then attempting to redirect the user to GitHub—this was failing with a CORS error. The error was limited to Safari on macOS and iOS1.
<!-- Working --> <link rel="stylesheet" href="/css/images.css"> <link rel="stylesheet" href="/css/article.css"> <!-- Not Working --> <link rel="stylesheet" href="/css/coder.min.32..Qs=" crossorigin="anonymous" media="screen"> <link rel="stylesheet" href="/css/coder-dark.min.e7...PY=" crossorigin="anonymous" media="screen">
The CSS files that weren’t working had a
crossorigin="anonymous" tag. According to MDN:
The “anonymous” keyword means that there will be no exchange of user credentials via cookies, client-side SSL certificates or HTTP authentication as described in the Terminology section of the CORS specification, unless it is in the same origin.
Unfortunately for me, there’s a bug in Safari where the
crossorigin="anonymous" setting refuses to send credentials even if it’s from the same origin… 😞 This bug was reported in 2017 and only fixed in May this year; it’s fixed in the Safari Technology Preview but not yet in the main version of Safari.
Sadly I will have to wait until that fix makes it in to production before I can access the dev version of my blog from my main browser.
Chrome worked on macOS but not on iOS presumably because it’s using the WebKit engine that all iOS browsers have to use. ↩︎