Welcome! This guide has 2 purposes:

• provide a reproducible workflow to developing a WASM filter for envoy

• Demonstrate how Envoy and WASM Filters can be used for edge computing applications

Warning: Envoy filter development is not for the faint of heart. This guide will give you a good intro and walk through the working example on Github. You may still encounter errors. The sources linked below may be helpful for debugging issues. Also, feel free to reach out!

This article is part of a Series on Edge Computing. For an introduction to edge computing see [the first article in the series](). For more guides like this, see the guide's home page.

The Problem

Before diving into writing an Envoy filter, let's define a real-world problem that we are going to try and solve.

Imagine we are a SaaS provider with a multi-tenant product running on Kubernetes. The problem is that the Ops team can't identify which customers are being impacted during incidents. The identified solution is to track the customer ID on each API request, allowing the Operations team to immediately know if the indcident is localized or affecting all customers. To update all the microservices to have this behavior was going to a big challenge. Instead, the business realized the Platform team could implement this solution inside their API Gateway and Service Mesh.

Since our company is running Istio, we can create an Envoy filter (more about this in the next section). This filter will look up the customer ID using the authorization header and update the response headers. Then envoy will add this custom header to the OpenTelemetry traces, allowing the Ops team to access the data they need.

Design

Before we implement the solution let's understand the key parts of envoy.

Envoy processes each request through a Listener. A listener defines a "chain" of filters, where each filter processes the request before the request is passed to the next filter. Most filters are written and compiled into Envoy. But one filter, the WASM filter, allows developers to write a custom envoy filter without having to compile Envoy.

The WASM filter accepts a WASM module and executes it for every request it processes. WASM modules are written with the proxy-wasm SDK. Let's get to writing our filter now.

Implement the Filter

We're going to develop an envoy filter in Rust. Rust has seen a lot of attention in the WebAssembly ecosystem. The Envoy project is no exception and we can expect fairly good support for rust. Consider cloning the Github repo, then using this guide as an in-depth explanation of the code.

Local Dev Setup

First, let's install Rust and Cargo, the Rust package manager/task runner. We'll use the language instructions here. The only additional step is that we need to install the WASM32 compilation target using the following command:

rustup update
rustup target add wasm32-unknown-unknown

Create The Project

Now we'll create a normal rust project using the Cargo CLI:

$ cargo new --lib envoy-filter-customer-lookup
     Created library `envoy-filter-customer-lookup` package

Next, we make 2 updates to Cargo.toml. Add the following dependencies:

log = "0.4.8"
proxy-wasm = "0.2.1"

We also need to tell Rust that this project will be compiled as a dynamic library:

[lib]
path = "src/lib.rs"
crate-type = ["cdylib"]

The full Cargo.toml file is on Github.

Code the Filter

Here is a "Hello World" envoy filter:

use log::info;
use proxy_wasm as wasm;

#[no_mangle]
pub fn _start() {
    proxy_wasm::set_log_level(wasm::types::LogLevel::Trace);
    proxy_wasm::set_http_context(
        |context_id, _root_context_id| -> Box<dyn wasm::traits::HttpContext> {
            Box::new(HelloWorld { context_id })
        },
    )
}

struct HelloWorld {
    context_id: u32,
}

impl wasm::traits::Context for HelloWorld {}

impl wasm::traits::HttpContext for HelloWorld {
    fn on_http_request_headers(&mut self, num_headers: usize, _end_of_stream: bool) -> wasm::types::Action {
        info!("Got {} HTTP headers in #{}.", num_headers, self.context_id);
        let headers = self.get_http_request_headers();
        let mut authority = "";

        for (name, value) in &headers {
            if name == ":authority" {
                authority = value;
            }
        }

        self.set_http_request_header("x-hello", Some(&format!("Hello world from {}", 
authority)));

        wasm::types::Action::Continue
    }
}

Here's a simple breakdown of the code:

• First, we create a "HelloWorld" struct to implement 2 traits against

• We will implement one method on wasm::traits::Context later, that's why we declared it

• The on_http_request_headers method gets called when the envoy filter has received the HTTP headers

• In this method, we grab the :authority header and use it to create a new header on the request

• Finally, our method returns wasm::types::Action::Continue to tell Envoy to keep processing the filter

Next, let's compile.

$ cargo build --target wasm32-unknown-unknown --release

Note: I got a error that looked like = note: xcrun: error: invalid active developer path (. This just meant that I needed to update XCode. Do that with xcode-select --install.

Deploy our filter inside Envoy

Let's test out the filter. Envoy provides docker based "Sandboxes," where there is a pre-configured envoy service that we can modify to suit our needs. I started with this sandbox. You can view the full configuration in Github here.

We'll update the docker-compose.yml to look like this:

services:
  proxy:
    image: envoyproxy/envoy:dev
    volumes:
    - ${PWD}/docker/envoy.yaml:/etc/envoy.yaml 
    - ${PWD}/target/wasm32-unknown-unknown/release/envoy_filter_customer_lookup.wasm:/lib/envoy_filter_customer_lookup.wasm
    depends_on:
    - web_service
    ports:
    - "8000:8000"
    command: /usr/local/bin/envoy -c /etc/envoy.yaml --service-cluster proxy -l info

  web_service:
    image: ealen/echo-server:0.7.0
    ports:
      - 80:80cp target/wasm32-unknown-unknown/release/envoy_filter_customer_lookup.wasm ./

This gets the Envoy container to mount our compiled WASM module.

The HTTP Filter configuration for Envoy looks like this:

          http_filters:
          - name: envoy.filters.http.wasm
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
              config:
                name: "my_plugin"
                root_id: "my_root_id"
                # if your wasm filter requires custom configuration you can add
                # as follows
                configuration:
                  "@type": "type.googleapis.com/google.protobuf.StringValue"
                  value: |
                    {}
                vm_config:
                  vm_id: "my_vm_id"
                  code:
                    local:
                      filename: "lib/envoy_filter_customer_lookup.wasm"

Don't forget to copy the compiled envoy filter to the local directory so that we docker can mount it:

cp target/wasm32-unknown-unknown/release/envoy_filter_customer_lookup.wasm ./

Bring up the docker containers and send a request:

docker compose up -d
curl localhost:8000

If we view the Envoy container's logs, we can see our WASM filter at work.

$ docker compose logs --follow
... 
proxy_1        | [2023-05-30 20:48:15.240][21][info][wasm] [source/extensions/common/wasm/context.cc:1148] wasm log my_plugin my_root_id my_vm_id: Got 8 HTTP headers in #2.
proxy_1        | [2023-05-30T20:48:15.239Z] "GET / HTTP/1.1" 200 - 0 627 36 35 "-" "curl/7.65.1" "ad277cc6-fb8b-4966-abc9-c6e8a739a069" "localhost:8000" "192.168.0.2:80"

Now we have a local development cycle to build our filter. Next, let's implement the solution that will send an HTTP request to the customer API and decorate the response headers with the customer ID.

Implement the Filter

For the sake of this guide, I built a simple Customer API. It's wired up in the docker-compose.yml and you can see the implementation here. In this section we'll wire up the customer API to our Envoy instance and then code the filter.

To allow our Envoy filter to communicate to it, we have to add an Envoy "cluster" to our config.

  - name: customer_api
    connect_timeout: 5s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: customer_api
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                protocol: TCP
                address: customer_api
                port_value: 3030

Now, let's code the filter. First, we must extract the auth header from the request headers, then issue an HTTP request to the customer API. We'll do this in the on_http_request_headers() method:

fn on_http_request_headers(&mut self, num_headers: usize, _end_of_stream: bool) -> wasm::types::Action {
        info!("Got {} HTTP headers in #{}.", num_headers, self.context_id);
        let headers = self.get_http_request_headers();
        let mut token = "";
        // grab the auth header
        for (name, value) in &headers {
            if name == ":Authorization" {
                token = value;
            }
        }
        // issue HTTP request to customer API
        let res = self.dispatch_http_call(
            "customer_api", 
            vec![
                (":method", "GET"),
                (":path", "/api/lookup"),
                (":authority", "customer_api"),
                ("Authorization", token)], 
            None, 
            vec![], Duration::from_millis(500)).unwrap();

        // stop further processing of this request
        wasm::types::Action::Pause
    }

Let's explain this code further. In Envoy, HTTP requests are issued via the dispatch_http_call() method. The method doesn't have a typical signature - this is one of the idiosyncrasies of developing in WebAssembly environments.

Finally, the on_http_request_headers() method returns wasm::types::Action::Pause, telling Envoy to not pass this request onto the next filter.

Next, we need to write some code to handle the response of the HTTP request we just issued. We will handle this in the on_http_call_response() method of the wasm::traits::Context trait. This method gets called whenever Envoy receives the response. Remember: our WASM code didn't origin the HTTP request, it made a request to the WASM host - the WASM filter - to send an HTTP request. The WASM host will then give our code the response, when it's available. Here's the code:

impl wasm::traits::Context for HelloWorld {
    fn on_http_call_response(&mut self, _token_id: u32, 
        _num_headers: usize, body_size: usize, _num_trailers: usize,) {
        info!("http call completed");
        match self.get_http_call_response_body(0, body_size) {
            Some(body) => {
                match std::str::from_utf8(&body) {
                    Ok(v) => {
                        info!("Body: {}", v);
                        let value = self.get_customer_id(v);
                        info!("customer id: {}", value);
                        self.customer_id = value;
                    }
                    Err(e) => info!("Invalid UTF-8 sequence: {}", e),
                };
            },
            None => info!("failed to get response body")
        }
        self.resume_http_request();

    }
}

There's a lot of nested pattern matching going on here, and that's simply to account for the possible failures scenarios. There may be a better way to code this in Rust. Here's a breakdown of the code:

• first, we request the HTTP body

• Next, we try to convert the byte steam to a UTF-8 string

• If that's successful, we call self.get_customer_id() which simply extracts the ID from the JSON response of the Customer API.

• We'll store this ID using self.customer_id = value, this ensures that we can access this customer ID at a later step

• Very important: at the end, we call self.resume_http_request() to ensure that envoy proceeds processing the request

Here's how we implemented the self.get_customer_id() method:

#[derive(Deserialize, Debug)]
struct CustomerLookupResponse {
    id: u32,
    name: String
} 
impl HelloWorld {
    fn get_customer_id(&mut self, raw: &str) -> String {
        let r: serde_json::error::Result<CustomerLookupResponse> = serde_json::from_str(raw);
        info!("parsed json: {:#?}", r);
        r.map_or(String::from("unknown"), |res|  res.id.to_string())
    }
}

This code simply attempts to marshal the raw HTTP response into the CustomerLookupResponse type, and if successful, return the id. Otherwise, it will return unknown. The last change necessary is to update put the customer ID in the response headers of the request.

To modify response headers, we'll use another method on wasm::traits::HTTPContext: on_http_response_headers().

    fn on_http_response_headers(&mut self, _num_headers: usize, _end_of_stream: bool) -> wasm::types::Action {

        self.set_http_response_header("x-customer-id", Some(&self.customer_id.as_str()));
        wasm::types::Action::Continue
    }

In this method, we're simply adding the header, and again, returning the Continue action. We've done it! Now let's try it out.

Test the Filter

If you haven't already done so, start the Envoy sandbox with docker compose up -d.

Compile our code into a WASM module with cargo build --target wasm32-unknown-unknown --release.

Now we can finally issue an HTTP request against our sandbox and see the x-customer-id header being added:

a-cusick@NFIT-cusick envoy-filter-customer-lookup % curl -i -H "Authorization: Bearer asdfzxcv" localhost:8000
HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
content-length: 619
etag: W/"26b-5e7bcwjiYoPrHjlpoapllr3LWCA"
date: Sat, 10 Jun 2023 08:18:27 GMT
x-envoy-upstream-service-time: 4
x-customer-id: 1
server: envoy

Violá!

The envoy logs are very helpful in understanding the behavior of your WASM filter:

In these logs, we can see the customer API response being properly marshaled into CustomerLookupResponse.

Summary

In the beginning, we established a business case where a SaaS business's Operations team needed to see the customer ID on all requests. This would allow the Ops team to instrument their monitoring tools to show which customers are being affected during an incident. We were able to accomplish this without changing any of the application code and simply leveraging Envoy, the proxy behind Istio.

The Envoy WASM SDK allows you to code WASM modules that interface with the WASM Filter. The WASM filter acts as a WASM Host, which calls methods in our WASM module - like on_http_call_response(), on_http_request_headers(). Our WASM module also makes calls the host, like dispatch_http_call().

Sources and Resources

• Envoy WASM Filter Docs

• Blog post on developing an envoy filter in rust

• Proxy-wasm crate

• Proxy-wasm docs