Writing an Istio WASM Plugin in Go for migrating 100s of services to new auth strategy (Part 2)

Part 2: Calling out to an external service

This article will mostly be about the building of the WASM plugin itself and less on the reasons why we needed it, but a 1-liner explanation is “We had to route our service-to-service traffic through an Nginx proxy to acquire an auth JWT from an Auth Service, and wanted to remove that extra network hop by having the WASM plugin call the Auth Service directly instead of Nginx”.

This post is all about calling out to another service in the service-mesh. To refer back to earlier parts use the links below:

• Part 1: Getting the basic project bootstrapped and running
• Part 2: Calling out to an external service
• Part 3: Configuring the plugin and accessing local container files
• Part 4: Metrics
• Part 5: Testing your WASM plugin
• Part 6: Gotchas

For this next part we’re going to make an HTTP request to another service that also resides in our Kubernetes cluster. This is a fairly common use-case (especially for auth), so we’ll fake this by using the Httpbin sample app to simulate an actual in-house auth service.

Remember you can follow along with code from https://github.com/henders/writing-an-envoy-wasm-plugin/tree/part2.

A loose tie-back to our overall purpose is to remove a network hop through Nginx just for the purpose of getting a JWT without which upstream services are not going to see any authentication JWT anymore. So our WASM plugin needs to help grab that JWT.

Photo by Patrick Robert Doyle on Unsplash

Building on the code from Part 1, we’re going to add the client code for calling an external service. To keep the code snippets under control, we’ll break this file into a few pieces.

First we’ll need a struct to wrap the request and response-callback for the Auth Service:

package internal

import (
  "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
  "time"
)

type AuthClient struct {
  XRequestID string
}

func (d *AuthClient) RequestJWT(origReqHeaders map[string]string) {
  proxywasm.LogInfof("%s: Requesting JWT from Auth Service", d.XRequestID)
  
  // Call the Auth Service
  _, err := proxywasm.DispatchHttpCall(
    "outbound|8000||httpbin.default.svc.cluster.local",
    [][2]string{
      {"accept", "*/*"},
      {":authority", "httpbin.default.svc.cluster.local"},
      {":method", "GET"},
      {":path", "/base64/SFRUUEJJTiBpcyBhd2Vzb21l"}, // get Httpbin to return a fake JWT
      {AuthHeader, origReqHeaders[AuthHeader]},      // Copy auth header from original request to auth against
    },
    nil, // no body necessary
    nil, // no trailers necessary
    150, // timeout is 150ms
    d.authCallback, // we'll define this callback function in a bit
  )
  if err != nil {
    proxywasm.LogCriticalf("%s: failed to call Auth Service: %v", d.XRequestID, err)
    // We want to resume the intercepted request even if we couldn't get an authentication header
    _ = proxywasm.ResumeHttpRequest()
  }
}

The important line is the DispatchHttpCall() call which does the actual HTTP request to our auth(Httpbin) service. I’ll describe the parameters a bit more:

• AuthClusterName: Note this name (“outbound|8000||httpbin.default.svc.cluster.local”) is not a standard “<DNS>:<Port>” format but instead is an Envoy-specific connection string. Any other service should be able to be contacted based on the base format “outbound|<port>||<k8s service name>.<k8s namespace>.svc.cluster.local”. You should be able to verify this by dumping all the known cluster-names from any envoy-sidecar like:

$ kubectl exec "$(kubectl get pod -l app=helloworld -o jsonpath='{.items[0].metadata.name}')" -c istio-proxy -- curl -sS 0:15000/clusters
outbound|8000||httpbin.default.svc.cluster.local::observability_name::outbound|8000||httpbin.default.svc.cluster.local
outbound|8000||httpbin.default.svc.cluster.local::default_priority::max_connections::4294967295
...
...

• headersToSend: This is just a straight key/value array of headers to send to the auth service. You’ll note that we had to set :authority to a normal type of FQDN. This is because outbound|8000||httpbin.default.svc.cluster.local is not a valid DNS that HTTP client/servers could handle. We override the :authority instead of host because Envoy uses HTTP/2 headers.
• callBack: This is the function that will be called back if we get a response. We can’t pass along any extra context parameters like XRequestID or similar, so that’s the reason we have encapsulated this RequestJWT and authCallback functions into the AuthClient struct. When the callback is called, it can access any member variables. We’d also need to do this if we wanted to track retries or other context that needed be propagated between calls.

Now we need to define the callback function specified in DispatchHttpCall:

func (d *AuthClient) authCallback(_, _, _ int) {
  proxywasm.LogInfof("%s: Got response from AuthService", d.XRequestID)
  
  // We want to always resume the intercepted request regardless of success/fail to avoid indefinitely blocking anything
  defer func() {
    if err := proxywasm.ResumeHttpRequest(); err != nil {
      proxywasm.LogCriticalf("%s: failed to ResumeHttpRequest after calling auth: %v", d.XRequestID, err)
    }
  }()
  
  // Get the response headers from our call to AuthService
  headers, err := proxywasm.GetHttpCallResponseHeaders()
  if err != nil {
    proxywasm.LogCriticalf("%s: failed to GetHttpCallResponseHeaders from auth response: %v", d.XRequestID, err)
    return
  }
  
  // Convert to map to make it easier to get specific headers
  authResponseHeaders := headerArrayToMap(headers)
  
  // Note we're using `:status` instead of just `status`. This is the same for any HTTP/2 headers like ':method', ':path', ':authority', ...
  // You don't need the ':' prefix for headers like 'user-agent', 'accept', ...
  if authResponseHeaders[":status"] == "200" {
    proxywasm.LogInfof("%s: AuthService gave successful (200) response", d.XRequestID)
    // Grab the response body in which Httpbin will return a fake JWT
    body, err := proxywasm.GetHttpCallResponseBody(0, 1024)
    if err != nil {
      proxywasm.LogCriticalf("%s: failed to GetHttpCallResponseBody for auth response: %v", d.XRequestID, err)
    }
  
    // Trim the trailing newline from the httpbin response body
    jwt := strings.Trim(string(body), "\r\n")
    // Now add this fake JWT into a new header to send to the upstream service
    proxywasm.LogInfof("%s: adding new header to original request: '%s=%s'", d.XRequestID, XAuthServiceAuthResponseJWTHeader, jwt)
    if err := proxywasm.AddHttpRequestHeader(XAuthServiceAuthResponseJWTHeader, string(body)); err != nil {
      proxywasm.LogCriticalf("%s: failed to add header '%v' to request: %v", d.XRequestID, XAuthServiceAuthResponseJWTHeader, err)
    }
    return
  }

  // If we got a non-200 response status, just log for now. This is where we'd add retries
  proxywasm.LogErrorf("%s: AuthService failed this request - status:%s", d.XRequestID, authResponseHeaders[":status"])
}

This callback needs to determine if the request failed based on the HTTP/2 header :status , and if the request for a JWT succeeded it’s going to add it to the originally intercepted request’s headers using AddHttpRequestHeader().

In both the request and callback functions you’ll note we always need to call ResumeHttpRequest() to ensure we don’t block this request forever and confuse both the client and service teams!

Next we’ll need to call this from our request handler by adding to the doSomethingWithRequest in request_handler.go:

func (r *RequestHandler) doSomethingWithRequest(reqHeaderMap map[string]string, xRequestID string) types.Action {
  // for now, let's just log all the request headers so we get an idea of what we have to work with
  for _, h := range reqHeaderMap {
    proxywasm.LogInfof("  %s: request header --> %s: %s", xRequestID, h[0], h[1])
  }
  
  // if auth header exists, call out to auth-service to request JWT
  if _, exists := reqHeaderMap[AuthHeader]; exists {
    authClient := AuthClient{XRequestID: xRequestID, Conf: r.Conf}
    authClient.RequestJWT(reqHeaderMap)
    // We need to tell Envoy to block this request until we get a response from the Auth Service
    return types.ActionPause
  }
  
  // If there was no authentication header to operate on, then
  // forward request to upstream service, i.e. unblock request
  return types.ActionContinue
}

Now we can recompile, build the docker image and deploy

$ tinygo build -o main.wasm -scheduler=none -target=wasi ./main.go
$ docker build -t wasmplugin .
$ docker push shender/wasmplugin:v2 # This path will obviously just work for me :)

We also now need to update the Kubernetes yaml to refer to the new version:

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: auth-wasm-plugin
  namespace: default
spec:
  imagePullPolicy: Always
  match:
    - mode: SERVER
  selector:
    matchLabels:
      app: helloworld
  url: oci://docker.io/shender/wasmplugin:v2

$ kubectl -n default apply -f k8s_deploy.yml

And we’ll push this to Kubernetes. Note that there is no downtime with code switches, the Envoy Sidecar will seamlessly load in the new WASM version without affecting any requests to the upstream service.

Let’s make another curl request:

$ kubectl exec "$(kubectl get pod -l app=helloworld -o jsonpath='{.items[0].metadata.name}')" -- curl -sS helloworld:5000/hello -H 'Authorization: totally fake'
Hello version: v1, instance: helloworld-v1-78b9f5c87f-v792v

We added the Authorization header just to trigger the external request to our auth-service (Httpbin).

You should see some extra logs this time around:

$ stern -n default -lapp=helloworld -c istio-proxy
...
...
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951555Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: WASM plugin Handling request thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951590Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> :authority: helloworld:5000 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951592Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> :path: /hello thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951593Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> :method: GET thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951595Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> :scheme: http thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951596Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> user-agent: curl/7.38.0 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951597Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> accept: */* thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951599Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> authorization: totally fake thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951600Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> x-forwarded-proto: http thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951605Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> x-request-id: 7651f6e9-52a8-957a-a71e-2ef67132c8e5 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951607Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> x-envoy-attempt-count: 1 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951609Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> x-b3-traceid: 4125809fc455334afb0f4177da26f0fa thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951610Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> x-b3-spanid: fb0f4177da26f0fa thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951612Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> x-b3-sampled: 1 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951613Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7651f6e9-52a8-957a-a71e-2ef67132c8e5: request header --> x-forwarded-client-cert: By=spiffe://cluster.local/ns/default/sa/default;Hash=0645eb33f73609fceb62bbe81d9ce25a182b6d9188e31fa9176b75effe726949;Subject="";URI=spiffe://cluster.local/ns/default/sa/default thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.951615Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 7651f6e9-52a8-957a-a71e-2ef67132c8e5: Requesting JWT: outbound|8000||httpbin.default.svc.cluster.local thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.956564Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 7651f6e9-52a8-957a-a71e-2ef67132c8e5: Got response from AuthService thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.956598Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 7651f6e9-52a8-957a-a71e-2ef67132c8e5: AuthService gave successful (200) response thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:44:27.956602Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 7651f6e9-52a8-957a-a71e-2ef67132c8e5: adding new header to original request: 'x-auth-jwt=FAKE_JWT' thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy [2023-06-01T22:44:27.951Z] "GET /hello HTTP/1.1" 200 - via_upstream - "-" 0 60 916 911 "-" "curl/7.38.0" "7651f6e9-52a8-957a-a71e-2ef67132c8e5" "helloworld:5000" "10.1.0.69:5000" inbound|5000|| 127.0.0.6:59735 10.1.0.69:5000 10.1.0.69:60706 outbound_.5000_._.helloworld.default.svc.cluster.local default
helloworld-v1-78b9f5c87f-v792v istio-proxy [2023-06-01T22:44:27.950Z] "GET /hello HTTP/1.1" 200 - via_upstream - "-" 0 60 918 917 "-" "curl/7.38.0" "7651f6e9-52a8-957a-a71e-2ef67132c8e5" "helloworld:5000" "10.1.0.69:5000" outbound|5000||helloworld.default.svc.cluster.local 10.1.0.69:60706 10.100.85.246:5000 10.1.0.69:44680 - default

Photo by Marc Najera on Unsplash

Big Success!

We can see from the line containing “AuthService gave successful (200)” that Httpbin successfully responded with a body containing our fake JWT which we duly add to the original request headers. Now when the deferred function ResumeHttpRequest() is called at the end of that response handler, Envoy will unblock the intercepted request and send it onto the helloworld service which will see our new header x-auth-response-jwt .

If any error occurs with talking to Httpbin it’ll still upstream the request to the helloworld service. If this is not what we want then we can short-circuit this request and respond directly back to the client with a 401 (or appropriate status code) by changing the response handler to exit the whole request early using SendHttpResponse(). For an example of this see the version of auth_client.go in the repo, a snippet showing it’s usage is:

func (d *AuthClient) authCallback(_, _, _ int) {
  ...
  ...

  // We want to resume the intercepted request if success, otherwise 
  // send a response directly back to client using SendHttpResponse
  defer func() {
    if responseStatus != 200 {
      responseErr := proxywasm.SendHttpResponse(responseStatus, [][2]string{{"generated-by", "My WASM plugin"}}, []byte("Failed to add JWT"), -1)
      if responseErr == nil {
        // Need to skip calling ResumeHttpRequest as we've already replied to the client now
        return 
      }
      proxywasm.LogErrorf("%s: failed to send %d back to client: %v", d.XRequestID, responseStatus, responseErr)
    }
    if err := proxywasm.ResumeHttpRequest(); err != nil {
      proxywasm.LogCriticalf("%s: failed to ResumeHttpRequest after calling auth: %v", d.XRequestID, err)
    }
  }()
  
  ...
  ...
}

The SendHttpResponse() function will send a response directly back to the client. It is important to note that you should not call ResumeHttpRequest() in this case as we don’t want the upstream service to process the request anymore.

It’s also a good idea to add a response header that identifies that this response was created by our WASM plugin, or upstream service teams might get a bit confused that the client got a response not generated by them.

In the next post we’ll look at configuration, as we might not want our auth-service cluster-name or timeouts hardcoded.