Open in app

Sign up

Sign in

Write

Sign up

Sign in

Writing an Istio WASM Plugin in Go for migrating 100s of services to new auth strategy (Part 1)

Shane Hender
Zendesk Engineering
Published in
9 min readJun 13, 2023

--

Computers heavily interconnected
Photo by Taylor Vick on Unsplash

This series will mostly be about the building of the WASM plugin itself and less on the reasons why we needed it, but a 1-liner explanation is “We had to route our service-to-service traffic through an Nginx proxy to acquire an auth JWT from an Auth Service, and wanted to remove that extra network hop by having the WASM plugin call the Auth Service directly instead of Nginx”.

Phew.. that was a mouthful, but let’s dig into the project. We’ll divide this up into 6 parts which we’ll publish over the subsequent weeks:

Part 1: Getting the basic project bootstrapped and running

Note that since Istio just wraps Envoy, nearly all of this is applicable to just creating a WASM plugin for Envoy, but we’ll use Istio’s functionality first where available.

What we’re working with:

Ok, let’s get started with setting up our dev environment so we can test our WASM plugin before going live in our Staging or Production environment. This is pretty important unless you feel isolated in post-Covid times and want the human interaction of people coming to you to ask why Staging is broken. 😅

House on fire representing staging

Assuming you have Kubernetes installed locally (via Docker, Kind, Minikube, …), just follow the Istio Getting Started guide and deploy two of the sample apps as well:

$ git clone git@github.com:istio/istio.git
$ cd istio
$ kubectl -n default apply -f istio/samples/helloworld/helloworld.yaml
$ kubectl -n default apply -f istio/samples/httpbin/httpbin.yaml

Now let’s create a very basic WASM plugin to log the headers on each request it intercepts. You can follow along with code from https://github.com/henders/writing-an-envoy-wasm-plugin/tree/part1

Create a go.mod file with a single dependency as we only need the WASM SDK to start with:

module envoyfilter

go 1.20

require github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0

Next let’s put together the minimum implementation to intercept request headers on any requests intercepted by the Envoy Sidecar.

package main

import (
  "envoyfilter/internal"
  
  "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
  "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
)

func main() {
  proxywasm.SetVMContext(&vmContext{})
}

type vmContext struct {
 // Embed the default VM context here,
 // so that we don't need to reimplement all the methods.
 types.DefaultVMContext
}

// NewPluginContext Override types.DefaultVMContext otherwise this plugin would do nothing :)
func (v *vmContext) NewPluginContext(contextID uint32) types.PluginContext {
  proxywasm.LogInfof("NewPluginContext context:%v", contextID)
  
  return &filterContext{}
}

type filterContext struct {
  // Embed the default plugin context here,
  // so that we don't need to reimplement all the methods.
  types.DefaultPluginContext
}

// OnPluginStart Override types.DefaultPluginContext.
func (h *filterContext) OnPluginStart(_ int) types.OnPluginStartStatus {
  return types.OnPluginStartStatusOK
}

// NewHttpContext Override types.DefaultPluginContext to allow us to declare a request handler for each
// intercepted request the Envoy Sidecar sends us
func (h *filterContext) NewHttpContext(contextID uint32) types.HttpContext {
  return &internal.RequestHandler{}
}

Most of that is boilerplate, and I’ve scattered some comments in there to help a bit. One thing to note is that while you can see contextIDs, they don’t seem to help determine any order of executions or help with debugging.

You’ll also notice a new object RequestHandler is needed, so let’s declare that now:

package internal

import (
  "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
  "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
)

type RequestHandler struct {
  // Bring in the callback functions
  types.DefaultHttpContext
}

const (
  XRequestIdHeader = "x-request-id"
)

// OnHttpRequestHeaders is called on every request we intercept with this WASM filter
// Check out the types.HttpContext interface to see what other callbacks you can override
//
// Note: Parameters are not needed here, but a brief description:
//   - numHeaders = fairly self-explanatory, the number of request headers
//   - endOfStream = only set to false when there is a request body (e.g. in a POST/PATCH/PUT request)
func (r *RequestHandler) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {
  proxywasm.LogInfof("WASM plugin Handling request")
  
  // Get the actual request headers from the Envoy Sidecar
  requestHeaders, err := proxywasm.GetHttpRequestHeaders()
  if err != nil {
    proxywasm.LogCriticalf("failed to get request headers: %v", err)
    // Allow Envoy Sidecar to forward this request to the upstream service
    return types.ActionContinue
  }
  
  // Convert the request headers to a map for easier access (more useful in subsequent sections)
  reqHeaderMap := headerArrayToMap(requestHeaders)
  
  // Get the x-request-id for grouping logs belonging to the same request
  xRequestID := reqHeaderMap[XRequestIdHeader]
  
  // Now we can take action on this request
  return doSomethingWithRequest(reqHeaderMap, xRequestID)
}

// headerArrayToMap is a simple function to convert from array of headers to a Map
func headerArrayToMap(requestHeaders [][2]string) map[string]string {
  headerMap := make(map[string]string)
  for _, header := range requestHeaders {
    headerMap[header[0]] = header[1]
  }
  return headerMap
}

func (r *RequestHandler) doSomethingWithRequest(reqHeaderMap map[string]string, xRequestID string) types.Action {
  // for now, let's just log all the request headers to we get an idea of what we have to work with
  for key,value := range reqHeaderMap {
    proxywasm.LogInfof("  %s: request header --> %s: %s", xRequestID, key, value)
  }

  // Forward request to upstream service, i.e. unblock request
  return types.ActionContinue
}

Last thing to get the WASM binary is to use TinyGo (because reasons) to compile it. This is probably the biggest restriction to using Go to develop WASM plugins, because the TinyGo compiler doesn’t support the full Go functionality, e.g. object reflection. We’ll go over this in the subsequent parts of this article, but for now, just execute:

$ tinygo build -o main.wasm -scheduler=none -target=wasi ./main.go

There should now be a main.wasm file in your main directory. Now we have the basic WASM plugin, let’s deploy this into Istio.

First we have to create a docker image to allow Envoy to grab the WASM binary from. So let’s create a dockerfile with the following contents:

# https://github.com/istio-ecosystem/wasm-extensions/blob/master/doc/how-to-build-oci-images.md
FROM scratch
ADD main.wasm ./plugin.wasm

This docker image has to be built like this, if you try to use Ubuntu or Alpine base image, Envoy will complain about the layer count. A quick build using Docker:

$ docker build -t wasmplugin .
$ docker push shender/wasmplugin:v1 # This path will obviously just work for me :)

We also need a K8s YML file to deploy this WASM Plugin:

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: auth-wasm-plugin
  namespace: default
spec:
  imagePullPolicy: Always
  match:
    - mode: SERVER
  selector:
    matchLabels:
      app: helloworld
  url: oci://docker.io/shender/wasmplugin:v1

The Istio WasmPlugin CRD is really just a wrapper over the EnvoyFilter CRD. The Envoy version has more customization options like serving the WASM binary from a Kubernetes ConfigMap. But we’ll proceed with using the Istio abstraction because it’s much easier to work with.

Most things should be straightforward here, but I’ll explain a few of the more arcane ones:

Mode

This allows us to tweak which type of traffic we want to intercept:

  • SERVER only intercepts inbound traffic to this service. This is what we want as we want to validate auth to the upstream service.
  • CLIENT only intercepts outbound traffic from this service
  • BOTH…pretty confident you can work that one out :)

Url

The url field is probably the hardest to get right as you have a few options to allow Envoy to grab your WASM binary:

  • You can deploy the WASM binary as part of the docker image that contains your service and then just reference it using file://... as a path to the location in the docker image.
  • You can host it using a webserver (nginx, …) and reference it using standard url like https://myserver.com/my_wasm_binary
  • Probably the best way is to distribute it from its own docker image. As you can see above I uploaded the docker image to docker-hub for this example, but if you use a private registry you have to deal with adding an imagePullSecret configuration to give Envoy permission to pull the image even if your Kubernetes cluster already has permission.

Before we deploy this, let’s enable more verbose logging on the Envoy Sidecar for WASM:

$ kubectl -n default exec "$(kubectl get pod -l app=helloworld -o jsonpath='{.items[0].metadata.name}')" -c istio-proxy pilot-agent request POST /logging"?wasm=info"

And to deploy it:

$ kubectl apply -f k8s_deploy.yml
wasmplugin.extensions.istio.io/auth-wasm-plugin created

You should see an initial slew of logs from our WASM Plugin:

$ stern -n default -lapp=helloworld -c istio-proxy
...
...
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:58.079997Z info wasm fetching image shender/wasmplugin from registry index.docker.io with tag v1
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:59.608401Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=17
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:59.615395Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:59.615788Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=17
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:59.616063Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:59.620375Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=17
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:59.624703Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=17
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:34:59.628778Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=17

Note that there are multiple lines even though this is a single Envoy Sidecar. This is because a WASM VM will be loaded in each Envoy Worker thread in the Envoy process.

Now let’s perform a curl request to the helloworld service:

$ kubectl exec "$(kubectl get pod -l app=helloworld -o jsonpath='{.items[0].metadata.name}')" -- curl -sS helloworld:5000/hello
Hello version: v1, instance: helloworld-v1-78b9f5c87f-v792v

Now you should see our request logging capture that request and output the headers:

helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.873923Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: WASM plugin Handling request thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874017Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> :authority: helloworld:5000 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874022Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> :path: /hello thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874023Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> :method: GET thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874025Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> :scheme: http thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874026Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> user-agent: curl/7.38.0 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874027Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> accept: */* thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874029Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> x-forwarded-proto: http thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874030Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> x-request-id: 7e78f1a5-6fa9-9a17-910f-0cfe586bacef thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874032Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> x-envoy-attempt-count: 1 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874033Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> x-b3-traceid: 005e72f668245ddf30044ec6aa1e10e0 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874035Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> x-b3-spanid: 30044ec6aa1e10e0 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874036Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> x-b3-sampled: 1 thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-01T22:36:20.874038Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   7e78f1a5-6fa9-9a17-910f-0cfe586bacef: request header --> x-forwarded-client-cert: By=spiffe://cluster.local/ns/default/sa/default;Hash=0645eb33f73609fceb62bbe81d9ce25a182b6d9188e31fa9176b75effe726949;Subject="";URI=spiffe://cluster.local/ns/default/sa/default thread=26
helloworld-v1-78b9f5c87f-v792v istio-proxy [2023-06-01T22:36:20.873Z] "GET /hello HTTP/1.1" 200 - via_upstream - "-" 0 60 962 961 "-" "curl/7.38.0" "7e78f1a5-6fa9-9a17-910f-0cfe586bacef" "helloworld:5000" "10.1.0.69:5000" inbound|5000|| 127.0.0.6:57607 10.1.0.69:5000 10.1.0.69:51310 outbound_.5000_._.helloworld.default.svc.cluster.local default
helloworld-v1-78b9f5c87f-v792v istio-proxy [2023-06-01T22:36:20.871Z] "GET /hello HTTP/1.1" 200 - via_upstream - "-" 0 60 969 969 "-" "curl/7.38.0" "7e78f1a5-6fa9-9a17-910f-0cfe586bacef" "helloworld:5000" "10.1.0.69:5000" outbound|5000||helloworld.default.svc.cluster.local 10.1.0.69:51310 10.100.85.246:5000 10.1.0.69:48162 - default

Congratulations, the basic setup is now running. In the next post we’ll add the ability to call out to another service (inside the Kubernetes cluster) to grab a JWT to add to our original request.

Success kid for first WASM plugin
Istio
Wasm
Envoy Filter
Go
Envoy

--

--

Shane Hender
Zendesk Engineering
Follow

Written by Shane Hender

65 Followers
Follow

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams