redventures/tfe_privatelinks_proxy repository overview

⁠privatelinks

The terraform code in this repository exists to help us configure PrivateLink services (producers) and endpoints (consumers) across our AWS accounts.

You can find much more documentation about the PrivateLink featureset in AWS's VPC documentation⁠, but we will summarize some of the ways we use PrivateLink to connect infrastructure at RedVentures and how to write Terraform code to accomplish them.

⁠PrivateLink, a quick introduction & justification

Remember VPC endpoints? Once upon a time, they were a way to route traffic destined for Amazon S3 through a private network instead of over the Internet, which promised to speed up your S3 transfers by maybe a lot.

PrivateLink is an extension of this concept that allows you to create custom VPC Endpoints for basically any service you want, provided, at least for now, that service communicates over TCP connections. This means, among other things, that you don't need to expose your service to traffic from the Internet just so share it with another VPC or another AWS account.

This differs from VPC Peering Connections⁠ and AWS Direct Connect⁠ in that PrivateLink only connects specific services and not entire networks. This crucial distinction has several advantages for Red Ventures. First, it allows us to continue creating VPCs in AWS without the need to avoid overlapping IP networks. Second, it helps us better track what is consuming which services. Third, it reduces (or can reduce) the need to involve multiple operational teams to set up a connection.

There are two key terms that you need to understand to read the Terraform code in this repository. The way this code uses them is unique. First, a service:

Then, on the other hand, an endpoint:

⁠Repository Contents

We mean this repository to be the primary place that we configure PrivateLink connections across the majority of our AWS accounts or at least those that our CLT-based Platform Engineering team manages. At time of writing, we have named the files in this repo by the location where its privatelink resources will reside. Most of these files reference a shortened version of an AWS account alias: e.g. datascience.tf holds PrivateLink resources for the rv-datascience AWS account.

⁠Adding new accounts

If you have a new account that was not part of our main RV AWS Organization as of May 2018, you will need to add some resources to it for the contents of this repository to work with that account.

Although this repository uses Terraform for all its features, we set up these AWS dependencies with a CloudFormation Stack Set, which we run from the rv-quicksilver AWS account. The template that we apply through this Stack Set also appears in this repo in the privatelink-cloudformation.yml file.

Adding a new account will require either adding your new account ID to the existing Stack Set or applying the template in this repo to that account.

⁠Terraform Modules

There are three local modules in this repository that you will use to make PrivateLink connections:

1. modules/endpoint - this is the module for PrivateLink consumers
2. modules/service - this is the module for PrivateLink producers which we can refer to by a static IP address
3. modules/hostname_service - this is the module for PrivateLink producers that we can only refer to by a hostname; we need extra infrastructure for these.

⁠module/endpoint

Steps to create an endpoint:

• Specify an aws provider specific for the account where the endpoint will be created
• Create a map variable that holds the vpc_id & account_id
• Get a reference (data object) to the vpc where the endpoint will be created
• Create a security group to control access
• Create the endpoint using: "modules/endpoint"

Example Invocation

⁠module/service

Example Invocation

⁠module/hostname_service and the nginx proxy container

If you want to create a PrivateLink connection with something that is behind another load balancer somewhere in one of our networks, you may need this module. For example, imagine that you need an application in our rv-hr AWS account to be able to make requests to jenkins-main.redventures.ninja, which runs in our rv-main AWS account. In this case, you likely cannoy rely on a static IP address because jenkins-main.redventures.ninja refers to an AWS load balancer whose IP addresses may change.

In this case, you would want to use this hostname_service module. This module will run a small nginx proxy container to help proxy traffic to the hostname of your choosing, whatever it may be.

If you need to debug this nginx proxy, you should inspect modules/hostname_service/proxy.tf and modules/hostname_service/nginx.conf.tpl to understand how we generate an nginx configuration for proxying. We run a small loader program, the code for which is in cmd/nginx_loader, that makes all environment variables available to a template file you specify. This program replaces template variables and writes an nginx configuration file to /etc/nginx/nginx.conf inside a container. The Dockerfile that builds this container is Dockerfile.nginx in the root of this repository.

We have set up automated builds on Docker Hub so that we build this image every time someone pushes a new commit or tag to this GitHub repo. The Docker URL for this image is here: https://hub.docker.com/r/redventures/tfe_privatelinks_proxy⁠

There is no private information in this nginx proxy container, so we have chosen to make this image publicly accessible to make using it across 80+ AWS accounts easier.

Example Invocation