Writing an Istio WASM Plugin in Go for migrating 100s of services to new auth strategy (Part 6)

Published in

Zendesk Engineering

4 min readAug 1, 2023

Part 6: Gotchas

This article will mostly be about the building of the WASM plugin itself and less on the reasons why we needed it, but a 1-liner explanation is “We had to route our service-to-service traffic through an Nginx proxy to acquire an auth JWT from an Auth Service, and wanted to remove that extra network hop by having the WASM plugin call the Auth Service directly instead of Nginx”.

This section is to inform you about any issues I had and potential solutions. To refer back to earlier parts use the links below:

Panics

Panics in the WASM binary will stall requests:

After putting an intentional index out of range error in the code and deploying, we make a request to helloworld to trigger it:

$ kubectl exec "$(kubectl get pod -l app=helloworld -o jsonpath='{.items[0].metadata.name}')" -- curl -sS helloworld:5000/hello -H 'Authorization: totally fake'
<stalled response>

We can see a response never comes unless we have a timeout configured. Also looking at the logs we can see the stack trace doesn’t really help us figure out where in our WASM binary where the issue it. This probably can’t be helped because there is no source-map back to the original Go code:

helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T22:23:22.226613Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: WASM plugin Handling request thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T22:23:22.226832Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   19299ad9-1681-975e-a832-767795a485fa: request header --> %!s(T=104): %!s(T=116) thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T22:23:22.226856Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   19299ad9-1681-975e-a832-767795a485fa: request header --> %!s(T=50): %!s(T=99) thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T22:23:22.226863Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   19299ad9-1681-975e-a832-767795a485fa: request header --> %!s(T=104): %!s(T=101) thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T22:23:22.226865Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   19299ad9-1681-975e-a832-767795a485fa: request header --> %!s(T=49): %!s(T=57) thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T22:23:22.226868Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: panic: runtime error: index out of range thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T22:23:22.229371Z error envoy wasm external/envoy/source/extensions/common/wasm/wasm_vm.cc:38 Function: proxy_on_request_headers failed: Uncaught RuntimeError: unreachable
helloworld-v1-78b9f5c87f-v792v istio-proxy Proxy-Wasm plugin in-VM backtrace:
helloworld-v1-78b9f5c87f-v792v istio-proxy   0:  0x74fa - runtime.runtimePanic
helloworld-v1-78b9f5c87f-v792v istio-proxy   1:  0xfb6 - runtime.lookupPanic
helloworld-v1-78b9f5c87f-v792v istio-proxy   2:  0x17083 - proxy_on_request_headers thread=25

Memory Footprint

The default GC that TinyGo provides is designed for small embedded devices where all the memory would be dedicated for a single purpose.

However this is not well suited for rolling out to Envoy Proxies as it’ll eat up all the available memory, starving the Envoy Host.

Thankfully there is a very simple (and hilariously named!) library we just need to add to our plugin: https://github.com/wasilibs/nottinygc

As you can see from the graph above, it basically fixed the GC for our plugin.

Accessing local files

By default the Envoy Sidecar in an Istio setup is sandboxed in the Kubernetes pod, and doesn’t have access to files it would normally be able to access on the local Kubernetes pod filesystem.

For one of our use-cases, the WASM plugin needed to access a secret stored in a file inside the Kubernetes pod’s filesystem. We first had to let Envoy access that file by adding an annotation to the Kubernetes pod itself:

"apiVersion": "apps/v1"
"kind": "Deployment"
"metadata":
  "name": "some-service"
  "namespace": "some-namespace"
"spec":
  "template":
    "metadata":
      "annotations":
        "sidecar.istio.io/userVolumeMount": "[{\"name\": \"some-volume\", \"mountPath\": \"/some_directory\"}]"
      "name": "some-service"
    "spec":
      ...
      ...

Then we had to modify the way Istio’s pilot-agent started up as that process bootstraps the Envoy proxy.

ISTIO_META_OUR_SECRET=$(cat /some_directory/our_secret) exec /usr/local/bin/pilot-agent "$@"

Now our WASM plugin can access this secret by:

ourSecret, err := proxywasm.GetProperty([]string{"node", "metadata", "OUR_SECRET"})

Needless to say this is pretty painful to have to update the global Istio startup script if a single pod wants to expose a file to a WASM plugin.

Docker Image References

You’d think that if the main Kubernetes cluster has been configured with tokens to access private docker image registries, then the Envoy sidecars should be able to pull it too., but unfortunately you have to another Kubernetes secret in each namespace you want your WASM Plugin to be loaded in. (linked in the WASMPlugin YML you deploy)

I also couldn’t figure out how to get Envoy to load my docker image from my local Docker Desktop registry that my Kubernetes cluster was running in, instead having to leverage Docker Hub just to even write a development guide. 😢

Go vs TinyGo

There are a few limitations with the main Golang compiler’s lack of support for non-Browser based WASM binaries that forced us to use TinyGo. We ran into some surprises:

Limited type-reflection: while we generally want to limit reflection for performance reasons, limited type-reflection makes using community libraries harder: most libraries utilize reflection in one form or another. On the bright side TinyGo is slowly improving this.
Unoptimized Maps: I use maps a lot (probably my favorite data structure), and it’s always on my mind that they haven’t been optimized yet in TinyGo.
Not all built-in Golang packages are supported, e.g. https://tinygo.org/docs/reference/lang-support/stdlib/#encodingjson, meaning we have to search for an alternative implementation in the broader community that doesn’t rely on reflection.