Keycloak w/ EKS + ALB (401 after auth)

Question

I’m currently trying to get Keycloak to run in EKS behind ALB and for the life of me, I can’t get it to work. I get the redirect to a login screen and after I log in - I instantly get presented with 401 Authorization required.

Infrastructure as follows: Browser (HTTPS) → ALB (HTTP between pods w/ OIDC config) → Target Group → Application in EKS

Keycloak with debug logs enabled doesn’t give me any errors at all:

keycloak logs

ALB on the other side gives me an AuthInvalidIdToken error after I sign in and it tries to redirect to the original URI:

2022-07-27T13:20:12.184736Z app/k8s-ingr-35696e2f02/3e62855cad3d2877 ip_address:53578 - -1 -1 -1 302 - 384 554 "GET https://application.domain.com:443/favicon.ico HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:769668795579:targetgroup/k8s-application-367cc81867/9fbca3b5630aecf1 "Root=1-62e13b8c-14d1e5670bc7d8d86056ddef" "application.domain.com" "arn:aws:acm:eu-west-1:769668795579:certificate/83bd567a-a624-4ff8-86e7-744ebff68575" 6 2022-07-27T13:20:12.184000Z "authenticate" "-" "-" "-" "-" "-" "-"
2022-07-27T13:20:12.314098Z app/k8s-ingr-35696e2f02/3e62855cad3d2877 ip_address:53578 10.0.2.19:8080 0.000 0.010 0.000 302 302 1509 3271 "GET https://keycloak.domain.com:443/realms/master/protocol/openid-connect/auth?client_id=oauth2-proxy&redirect_uri=https%3A%2F%2Fapplication.domain.com%2Foauth2%2Fidpresponse&response_type=code&scope=email%20openid&state=Giz6L1tOQP4KTWIT%2BvrEvnRV3tnHFhCmswD%2B2tattLy8%2FRThp1hYyCrNLQQdG%2FIWq2JjO1p8rjDMHawgK0JW3S%2BszvY6JrZ9%2BeMzBmp05iTDXRre2IWaYnYxlH86i1FVCYrkIONZ0bdxu0UydsLuXEhx57FGsu6fbjKMNxAKO2cKPvDn7xZYLzCQuJWKKx3NZXUSEVNGgQLtCLMTM9YRBA%3D%3D HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:769668795579:targetgroup/k8s-keycloak-1f3317ccac/0ac746adee6d293e "Root=1-62e13b8c-73b50ba974b867b243093ce4" "application.domain.com" "arn:aws:acm:eu-west-1:769668795579:certificate/83bd567a-a624-4ff8-86e7-744ebff68575" 11 2022-07-27T13:20:12.303000Z "authenticate,forward" "-" "-" "10.0.2.19:8080" "302" "-" "-"
2022-07-27T13:20:12.479301Z app/k8s-ingr-35696e2f02/3e62855cad3d2877 ip_address:53578 - -1 -1 -1 401 - 674 616 "GET https://application.domain.com:443/oauth2/idpresponse?state=Giz6L1tOQP4KTWIT%2BvrEvnRV3tnHFhCmswD%2B2tattLy8%2FRThp1hYyCrNLQQdG%2FIWq2JjO1p8rjDMHawgK0JW3S%2BszvY6JrZ9%2BeMzBmp05iTDXRre2IWaYnYxlH86i1FVCYrkIONZ0bdxu0UydsLuXEhx57FGsu6fbjKMNxAKO2cKPvDn7xZYLzCQuJWKKx3NZXUSEVNGgQLtCLMTM9YRBA%3D%3D&session_state=f0c7b40f-df70-40a4-a4ec-ae674eddf704&code=5e30afcf-2121-4c3f-a2c5-6be9ccce569e.f0c7b40f-df70-40a4-a4ec-ae674eddf704.398968b2-cc51-4f41-8680-92e32051d6b0 HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 - "Root=1-62e13b8c-466c6d524d1bd9a745476851" "application.domain.com" "arn:aws:acm:eu-west-1:769668795579:certificate/83bd567a-a624-4ff8-86e7-744ebff68575" -1 2022-07-27T13:20:12.459000Z "authenticate" "-" "AuthInvalidIdToken" "-" "-" "-" "-"

ALB annotations for OIDC auth (Keycloak) is nothing fancy and/or custom:

auth_path = f"https://{self.keycloak_domain}/realms/{self.keycloak.realm}"
auth_annotations = base_annotations | {
    "alb.ingress.kubernetes.io/auth-type": "oidc",
    "alb.ingress.kubernetes.io/auth-idp-oidc": (
        f'{{"issuer":"https://{self.keycloak_domain}/",'
        f'"authorizationEndpoint":"{auth_path}/protocol/openid-connect/auth",'
        f'"tokenEndpoint":"{auth_path}/protocol/openid-connect/token",'
        f'"userInfoEndpoint":"{auth_path}/protocol/openid-connect/userinfo",'
        f'"secretName":"{self.keycloak.secrets_name}"}}'
    ),
    "alb.ingress.kubernetes.io/auth-on-unauthenticated-request": "authenticate",
}

ALB route in AWS

Keycloak config:

fullnameOverride: "{{ full_name }}"
extraStartupArgs: "--spi-login-protocol-openid-connect-legacy-logout-redirect-uri=true"

service:
  type: "ClusterIP"

auth:
  createAdminUser: true
  adminUser: "{{ admin_username }}"
  adminPassword: "{{ admin_password }}"

postgresql:
  enabled: false

externalDatabase:
  host: "{{ postgresql_host }}"
  database: "{{ postgresql_db }}"
  user: "{{ postgresql_user }}"
  password: "{{ postgresql_password }}"

extraEnvVars:
  - name: KEYCLOAK_HOSTNAME
    value: "{{ hostname }}"
  - name: PROXY_ADDRESS_FORWARDING
    value: "true"

What I’ve tried:

1. KEYCLOAK_FRONTEND_URL
2. Adding a trailing / to URI’s
3. Setting ALB session cookie names (KEYCLOAK_SESSION for example)
4. Enforcing HTTPS for ALB internal routing (alb_backend_protocol annotation)
5. Running everything with redirect_uri in Keycloak as *
6. List item

Some options I’ve come up while trying to debug this:

AWS ALB parses keycloak auth response somehow wrong? Misconfiguration from my side? (even tho keycloak is happy with everything?)

Versions:

Keycloak (bitnami) Helm Chart: 9.2.2 chart w/ 18.0.0 Keycloak

AWS ALB Helm Chart: 1.4.2 chart w/ 2.4.2 ALB

Answer 1

Turns out that according to openid connect you have to validate authorization_code according to these steps: https://openid.net/specs/openid-connect-basic-1_0.html#IDTokenValidation

What my issue here was is that the issuer in my OIDC config != iss claim from id_token. Basically changing:

f'{{"issuer":"https://{self.keycloak_domain}/",' -> f'{{"issuer":"https://{self.keycloak_domain}/realms/master",'

fixed the issue!

also turns out that oauth2-proxy doesn't validate user claims from id_token