gremlin/envoy-plugin repository overview

⁠gremlin/envoy-plugin

Integrate your Service Mesh with the Gremlin Failure Flags Platform for application-level fault injection via network middleware.

⁠Overview

This project is an Envoy HTTP filter extension that injects user-defined failures and payload mutation based on Gremlin experiments. This is a beta revision, with corresponding installation and configuration options.

⁠What does this project actually do?

This provides an HTTP Filtering Envoy Plugin integrated with the Gremlin Failure Flags platform. Adding it to sidecar proxies will instrument inbound and outbound HTTP connections as Failure Flags for the attached service providing:

1. Application-level targeting instrumentation
2. Network fault injection capabilities
3. Mesh topology mapping

⁠Is it safe?

Gremlin designs its products to be safe for production environments. This and other components are designed to fail safe and operate outside of the critical path for your appliciations and services.

⁠Installation

The following steps will walk the reader through installing the Gremlin Envoy Plugin into an existing Kubernetes cluster running Isio 1.22 in Sidecar mode.

⁠1. Add Configuration for the Gremlin Envoy Plugin

The first step adds your Gremlin agent credentials into a Secret in your target Kubernetes cluster so the Plugin can query the Gremlin API for experiment metadata. Replace the placeholders in this code snippet with values for your environment and account.

apiVersion: v1
kind: Secret
metadata:
  name: gremlin-service-mesh-config
  namespace: <namespace>
type: Opaque
stringData:
  config.yaml: |
    gremlin_team_certificate: <base64-encoded>
    gremlin_team_id: <base64-encoded>
    gremlin_team_private_key: <base64-encoded>

⁠2. Add the Gremlin Envoy Plugin to your Envoy Sidecars

Second, you'll need to install the plugin into the target PodSpec. This requires two changes.

⁠Add configuration annotations to the target PodSpec

These annotations tell Istio how to find and configure the plugin.

sidecar.istio.io/userVolumeMount: '[{"name":"gremlin-plugin","mountPath":"/gremlin/filter"}, {"name":"gremlin-service-mesh-config","mountPath":"/gremlin/config"}]'
sidecar.istio.io/userVolume: '[{"name":"gremlin-service-mesh-config", "secret":{"secretName":"gremlin-service-mesh-config"}}]'
sidecar.istio.io/logLevel: 'info'

⁠Add a volume to the target pod specs

This volume will be used by the program in the next step to make the plugin available to the proxy sidecar.

 volumes:
 - name: gremlin-plugin
 emptyDir: { }

⁠Add an Init Container to the target pod specs

This Init Container uses the gremlin/envoy-plugin and moves the plugin into place for Envoy.

initContainers:
- name: install-gremlin-plugin
  image: docker.io/gremlin/envoy-plugin:latest #v0.5 or envoy-1.30
  imagePullPolicy: Always
  volumeMounts:
    - name: gremlin-plugin
      mountPath: /gremlin/filter

⁠3. Add a new EnvoyFilter Resource

Add the following resource after substituting values appropriate for your environment.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gremlin-http-filter
  namespace: <namespace>
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: ANY
        listener:
          portNumber: <port where the service is exposed>
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.golang
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.http.golang.v3alpha.Config"
            library_id: gremlin-http-filter
            library_path: "/gremlin/filter/gremlin-http-filter.so"
            plugin_name: gremlin-http-filter
            plugin_config:
              "@type": type.googleapis.com/xds.type.v3.TypedStruct
              value:
                region: <your region> # required
                cloud: <name of your cloud> # required
                gremlin_debug: true # (default: false)

After these changes have been made you should be able to see logs for this plugin in your Envoy sidecar pod logs. If everything worked correctly you should also see the service you've added the plugin to show up in the Failure Flags panel in the Gremlin app.