Writing an Istio WASM Plugin in Go for migrating 100s of services to new auth strategy (Part 3)

Published in

Zendesk Engineering

7 min readJun 27, 2023

Part 3: Configuring the plugin

This article will mostly be about the building of the WASM plugin itself and less on the reasons why we needed it, but a 1-liner explanation is “We had to route our service-to-service traffic through an Nginx proxy to acquire an auth JWT from an Auth Service, and wanted to remove that extra network hop by having the WASM plugin call the Auth Service directly instead of Nginx”.

This section is all about adding configurability to your plugin. To refer back to earlier parts use the links below:

Part 1: Getting the basic project bootstrapped and running
Part 2: Calling out to an external service
Part 3: Configuring the plugin and accessing local container files
Part 4: Metrics
Part 5: Testing your WASM plugin
Part 6: Gotchas

Earlier, we hardcoded the connection details to our (fake) auth service. Now we’ll show how to configure it based on content from the WASMPlugin k8s CRD itself.

The Istio WasmPlugin YML supports a configuration section call pluginConfig . Let’s add our Auth configuration settings to this section:

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: auth-wasm-plugin
  namespace: default
spec:
  imagePullPolicy: Always
  match:
    - mode: SERVER
  pluginConfig:
    auth_authority: httpbin.default.svc.cluster.local
    auth_cluster_name: outbound|8000||httpbin.default.svc.cluster.local
    auth_timeout_ms: 150
  selector:
    matchLabels:
      app: helloworld
  url: oci://docker.io/shender/wasmplugin:v3

The pluginConfig section is documented in Istio as a freeform struct. All this means is that we can put any valid YML in this section and receive it in our WASM binary as it’s JSON equivalent.

This is where TinyGo limitations start affecting us with its limited support for type reflection as we can’t use Go’s built-in JSON marshaling due to its dependence on reflection. So we had to leverage an open source library, github.com/tidwall/gjson, that didn’t depend on it.

Using the code below we’ll load these settings into a Config struct in case we want to call a different service DNS or have a different timeout. I won’t explain the code too much because the only API we’re using is GetPluginConfiguration(), the rest is generic gjson library usage:

package internal

import (
  "os"
  "time"
  
  "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
  "github.com/tidwall/gjson"
)

const (
  AuthTimeoutDefault = time.Second
)

// Config is used to extract any WASMPlugin configuration defined in the deployed YML
type Config struct {
  AuthClusterName string
  AuthAuthority   string
  AuthTimeout     uint32
}

func NewConfig() *Config {
  configuration := getPluginConfiguration()
  config := Config{
    AuthClusterName: getStringFromConfig(configuration, "auth_cluster_name"),
    AuthAuthority:   getStringFromConfig(configuration, "auth_authority"),
    AuthTimeout:     uint32(getInt64FromConfig(configuration, "auth_timeout_ms", AuthTimeoutDefault.Milliseconds())),
  }
  
  return &config
}

func getPluginConfiguration() gjson.Result {
  proxywasm.LogInfof("Getting WASM plugin config...")
  configuration, err := proxywasm.GetPluginConfiguration()
  if err != nil {
    proxywasm.LogCriticalf("error reading plugin configuration: %v", err)
  }
  if len(configuration) == 0 {
    proxywasm.LogCritical("WASM plugin config was empty")
    return gjson.Result{}
  }
  if !gjson.ValidBytes(configuration) {
    proxywasm.LogCriticalf("WASM plugin config was invalid: %s", configuration)
    return gjson.Result{}
  }
  
  result := gjson.ParseBytes(configuration)
  return result
}

func getStringFromConfig(configuration gjson.Result, key string) string {
  result := configuration.Get(key)
  if result.Exists() {
    return result.String()
  }
  proxywasm.LogCriticalf("Configuration for '%s' wasn't set in config:%s", key, configuration)
  return ""
}

func getInt64FromConfig(configuration gjson.Result, key string, defaultResult int64) int64 {
  result := configuration.Get(key)
  if result.Exists() {
    return result.Int()
  }
  proxywasm.LogCriticalf("Configuration for '%s' wasn't set in config:%s", key, configuration)
  return defaultResult
}

Another restriction we’re under is that we can only call GetPluginConfiguration() from one location in our WASM Plugin and that is during the OnPluginStart callback in main.go.

Let’s go back to main.go and add support for reading the configuration during startup:


type filterContext struct {
  ...
  ...
  
  // Adding a reference to the PluginConfig to our main Context
  conf *internal.Config
}

func (h *filterContext) OnPluginStart(_ int) types.OnPluginStartStatus {
  // Modifying OnPluginStart to read the PluginConfig from Envoy Host
  h.conf = internal.NewConfig()
  proxywasm.LogInfof("Config loaded: %+v", h.conf)
  
  return types.OnPluginStartStatusOK
}

After that just pass this Config object all the way down to the AuthClient . This is straightforward Go code, so refer to the sample repo if you’d like to see the implementation.

Now when we deploy this version to Kubernetes, we should see logs similar to:

helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:39:27.986825Z info wasm fetching image shender/wasmplugin from registry index.docker.io with tag v3
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:39:28.007093Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: NewPluginContext context:1 thread=17
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:39:28.007256Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: Getting WASM plugin config... thread=17
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:39:28.007489Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log: Config loaded: {AuthClusterName:outbound|8000||httpbin.default.svc.cluster.local AuthAuthority:httpbin.default.svc.cluster.local AuthTimeout:150} thread=17

And making another HTTP request to helloworld should result in the same logs as Part 2:

$ kubectl exec "$(kubectl get pod -l app=helloworld -o jsonpath='{.items[0].metadata.name}')" -- curl -sS helloworld:5000/hello -H 'Authorization: totally fake'
Hello version: v1, instance: helloworld-v1-78b9f5c87f-v792v

$ stern -n default -lapp=helloworld -c istio-proxy
...
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705896Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: WASM plugin Handling request thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705936Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> :authority: helloworld:5000 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705939Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> :path: /hello thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705940Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> :method: GET thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705941Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> :scheme: http thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705943Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> user-agent: curl/7.38.0 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705944Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> accept: */* thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705946Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> authorization: totally fake thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705947Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> x-forwarded-proto: http thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705949Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> x-request-id: 1cf17a7b-77b8-9104-b37b-037df7009805 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705950Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> x-envoy-attempt-count: 1 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705952Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> x-b3-traceid: fd84c29d925f6e2c9a1b801526dcce34 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705953Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> x-b3-spanid: 9a1b801526dcce34 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705954Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> x-b3-sampled: 1 thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705964Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin:   1cf17a7b-77b8-9104-b37b-037df7009805: request header --> x-forwarded-client-cert: By=spiffe://cluster.local/ns/default/sa/default;Hash=0645eb33f73609fceb62bbe81d9ce25a182b6d9188e31fa9176b75effe726949;Subject="";URI=spiffe://cluster.local/ns/default/sa/default thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.705974Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 1cf17a7b-77b8-9104-b37b-037df7009805: Requesting JWT from 'outbound|8000||httpbin.default.svc.cluster.local' with headers:[[accept */*] [:authority httpbin.default.svc.cluster.local] [:method GET] [:path /base64/RkFLRV9KV1QK] [authorization totally fake]] thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.724143Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 1cf17a7b-77b8-9104-b37b-037df7009805: Got response from AuthService thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.724184Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 1cf17a7b-77b8-9104-b37b-037df7009805: AuthService gave successful (200) response thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy 2023-06-02T00:45:07.724190Z info envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1148 wasm log default.auth-wasm-plugin: 1cf17a7b-77b8-9104-b37b-037df7009805: adding new header to original request: 'x-auth-jwt=FAKE_JWT' thread=25
helloworld-v1-78b9f5c87f-v792v istio-proxy [2023-06-02T00:45:07.705Z] "GET /hello HTTP/1.1" 200 - via_upstream - "-" 0 60 1028 1010 "-" "curl/7.38.0" "1cf17a7b-77b8-9104-b37b-037df7009805" "helloworld:5000" "10.1.0.69:5000" inbound|5000|| 127.0.0.6:60291 10.1.0.69:5000 10.1.0.69:55026 outbound_.5000_._.helloworld.default.svc.cluster.local default
helloworld-v1-78b9f5c87f-v792v istio-proxy [2023-06-02T00:45:07.705Z] "GET /hello HTTP/1.1" 200 - via_upstream - "-" 0 60 1029 1029 "-" "curl/7.38.0" "1cf17a7b-77b8-9104-b37b-037df7009805" "helloworld:5000" "10.1.0.69:5000" outbound|5000||helloworld.default.svc.cluster.local 10.1.0.69:55026 10.100.85.246:5000 10.1.0.69:41606 - default

Sometimes we also want to know the Kubernetes namespace we’re running in for extra logging/metric information, in which case we can leverage the VmConfig. This allows us to either specify explicit values for ENV Vars (though this is probably not that useful if you are already leveraging the PluginConfig functionality), or you can request ENV Vars from the Envoy Host to be accessible from within the WASM plugin.

We can ask for these Envoy Host environment variables by adding to the VmConfig setting:

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: auth-wasm-plugin
  namespace: default
spec:
  imagePullPolicy: Always
  match:
    - mode: SERVER
  pluginConfig:
    auth_authority: httpbin.default.svc.cluster.local
    auth_cluster_name: outbound|8000||httpbin.default.svc.cluster.local
    auth_timeout_ms: 150
  selector:
    matchLabels:
      app: helloworld
  url: oci://docker.io/shender/wasmplugin:v3
  vmConfig:
    env:
      - name: POD_NAMESPACE
        valueFrom: HOST

This is telling the Envoy Host that loads our WASM binary to also make available its POD_NAMESPACE ENV var. To access this, we can call the Go standard function:

if namespace, exists := os.LookupEnv("POD_NAMESPACE"); exists {
  return namespace
}

In fact, we could use ENV vars for everything in this example if we wished:

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: auth-wasm-plugin
  namespace: default
spec:
  imagePullPolicy: Always
  match:
    - mode: SERVER
  selector:
    matchLabels:
      app: helloworld
  url: oci://docker.io/shender/wasmplugin:v3
  vmConfig:
    env:
      - name: POD_NAMESPACE
        valueFrom: HOST
      - name: AUTH_AUTHORITY
        value: httpbin.default.svc.cluster.local
      - name: AUTH_CLUSTERNAME
        value: outbound|8000||httpbin.default.svc.cluster.local
      - name: AUTH_TIMEOUT
        value: 150

But this does limit you to a flat list of settings, rather than potentially having related configuration nested together in the YML, or array/object type configurations.

In the next post we’ll show you how to add the all important telemetry to your plugin.

Writing an Istio WASM Plugin in Go for migrating 100s of services to new auth strategy (Part 3)

Written by Shane Hender