kayako/pripy repository overview

⁠PriPy

Maintain simple private pip repository on S3.

⁠Serving Packages

PriPy can serve python packages from an S3 backed API gateway. To set up the private repository you'll need to create a lambda function and make it accessible via API gateway.

The lambda function lists the packages and builds available on S3 and a build can be downloaded directly by the API gateway.

⁠Setup

⁠Command line tool

To install the command line utility pripy simple install it with pip

pip install --extra-index-url http://pripy.kayakodev.net/simple/ pripy

and you should have pripy installed on your machine

pripy --help

⁠Repository

Setting up the repository consists of 3 steps

• Setting up S3 bucket
• Setting up Lambda function
• Setting up API gateway
• (Optional) Setting up a custom domain

Setting up the S3 bucket is easy. Just create a new bucket using S3 console or aws cli

aws --profile <profile> s3 mb pripy.bucket_name

Now create a new IAM role with following policies

• AWSLambdaBasicExecutionRole
• AWSLambdaS3ExecutionRole
• Following custom policy to allow this role to list and get objects from S3 bucket

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:ListBucket"
            ],
            "Resource": [
                pripy.py,
                pripy.py
            ]
        },
        {
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "*"
        }
    ]
}

After creating the role add following trust relationship configuration

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

This trust relationship lets Lambda and API gateway assume the role.

Save the ARN of the role as it will be required during API Gateway configuration.

⁠Lambda function

The lambda function will be used to list all available packages and their builds. This function will also be responsible to authenticate the user request for us. The function ships with a very simple authentication provider which fetches the list of users and their password from S3 and tries to authenticate the request with it. However, authentication is completely optional and can be turned off if you want the repository to be publicly accessible.

⁠Setting up the function

Create a new Lambda function with the IAM role created in previous step. Feel free to choose any name, the example here will assume the name to be PriPy. Zip __init__.py, api.py file and auth.py and upload it to Lambda, on MacOS X you can do

zip lambda.zip __init__.py api.py auth.py

to generate the zip file.

Configure the lambda handler function to api.handle_lambda.

Finally, add following environment variables

You may also want to make sure that the bucket which contains the credentials is properly secured and lambda function has permission to read the file.

⁠API Gateway

API Gateway is responsible for authenticating the requests with authentication provider, invoking the Lambda function to list available packages and builds and finally serving builds from S3.

The Gateway provides three API endpoints in total.

• GET /simple/
• GET /simple/{package}
• GET /simple/{package}/{build}

To set up the API gateway, go to the API gateway console and create a new API set. The example here assumes the name to be pripy.

Inside the API, create three new resources

• /simple/
• /simple/{package}/
• /simple/{package}/{build}

Path parameter names are important to get right as the lambda function depends on the name of parameters.

⁠Configuration

Let's setup the methods on these API resources now.

⁠simple resource

Create a GET method on /simple resource and configure it as follows

After saving the method, click on Method Request and add a header Authorization.

⁠packages resource

Create GET method on /simple/{package} and configure it as follows

After saving the method, click on Method Request and add a header Authorization.

⁠build resource

Create GET method on /simple/{package}/{build} and configure it as follows

We are using a stage variable named bucket to route the requests to S3 bucket, this variable will be configured when the API is deployed.

Now go to Integration Request and configure the URL Path Parameters as follows:

Now go back and go to Method Response, expand the row for HTTP Status 200 and add following Response Headers for 200

• Timestamp
• Content-Type
• Content-Length

Go back again and go to Integration Response, expand the row for method response status 200, set Content handling to Convert to binary (if needed) and add following header mappings:

With API endpoints all configured, let's setup the binary support so that we can download the files from S3 via API gateway.

Select Binary Support from left most pane and add content types that you want to be able to download. pripy can push builds in all formats supported by distutil so it is sensible to mark all these formats as binary content.

• application/x-tar
• application/x-binary
• application/octet-stream
• application/zip
• binary/octet-stream

⁠Authentication

The API gateway is now ready to use. The last step is to set up the authentication.

Select pripy API from left most pane and click on Authorizers under it. Configure the authorizer as follows:

Now enable authentication by following these steps:

1. Go to pripy API
2. Select GET /simple/ method
3. Go to Method Request
4. Edit Authorization and select the custom authorizer
5. Save and repeat step 2 through 4 for remaining two methods

⁠S3 Authentication

If you configured the API authentication to use S3 as a backend, make sure the file that is configured to provide credentials exists and contains valid JSON object.

Keys in the JSON object are user names and their corresponding values are the passwords. Example:

{
  "user_one": "randompassword",
  "user_two": "anotherpassword"
}

⁠Deployment

API Gateway and Lambda function is now ready for deployment.

Go to API Gateway and select pripy. From actions menu select Deploy API and create a new stage. The example assumes the stage name to be beta.

When the stage is created, select Stage Variables tab and add the stage variable:

⁠Custom domain

You can optionally configure a custom domain name with SSL to serve the repository.

In order to configure a custom domain name you must import the SSL certificate for the domain you want to use into AWS Certificate Manager. You can also request a fresh certificate if you don't have one already.

Select Custom Domain Names from left panel and select Create Custom Domain Name.

Configure the domain name as follows

Copy the Distribution Domain Name that was just generated and configure the DNS records for the domain to point to this distribution.

⁠Managing packages

PriPy also provides a handy command line utility pripy. The command line can do various tasks to ease the management. For more details please see

pripy --help

It is thoroughly documented and up to date will current state of codebase.

⁠Requirements

A package which is to be maintained by PriPy must fulfil following requirements:

• Setup configuration must be done in setup.cfg file. This is a good idea to follow in general because it makes the setup configuration consumable by external tools. PriPy reads the package information from setup.cfg file and fails if the file could not be found.
• The package must follow basic semantic version scheme. It must declare the version in setup.cfg file as major.minor.patch string. Extended version information such as beta or rc are not supported right now.

⁠Available Commands

Following commands are available in pripy command line tool:

⁠Using With PIP

Installing packages with pip is fairly simple. The quickest way is to run

pip install --extra-index-url https://username:[email protected]/simple package_name

Where the URL to API gateway must be substituted for example URL and package_name would be the package you want to install. Username and password can be omitted if authentication is not set up.

Note that the path /simple in the URL is important otherwise pip will not be able to discover any package.

To permanently configure pip to use this repository you must edit ~/.pip/pip.conf and include following:

[install]
extra-index-url = https://username:[email protected]/simple

Now pip will by default look into this repository every time you try to install a package with

pip install package_name