heynemann/fastlane repository overview

⁠fastlane⁠ service

fastlane⁠ is a redis⁠-based queueing service that outsmarts everyone else by using containers.

More seriously, though, fastlane⁠ allows you to easily implement new workers in the form of containers.

Instead of the tedious, repetitive work of yesteryear where you had to implement a worker in language X or Y, you just spin a new container with all the dependencies you require already previously installed, and instruct fastlane⁠ to run a command in that container. Bang! Instant Super-Powered Workers!

⁠Features

• Ad-Hoc execution of jobs (run job right now)⁠
• Scheduled execution of jobs (run job next sunday at 6am, or run in 10 minutes from now)⁠;
• Crontab execution of jobs (run job at */10 * * * * - every ten minutes)⁠;
• Allows job details to be updated⁠;
• API to retrieve tasks, /tasks⁠;
• API to retrieve task details, /tasks/my-task (<taskUrl>);
• API to retrieve job details, <taskUrl>/jobs/<jobId> (<jobUrl>);
• API to stop running job (<jobUrl>/stop);
• API to retry job (<jobUrl>/retry);
• API to get logs(<jobUrl>/logs), stdout (<jobUrl>/stdout) and stderr (<jobUrl>/stderr) for last execution in jobs;
• Job log output streaming using WebSockets (ws://<jobUrl>/ws) and <jobUrl>/stream;
• API to retrieve execution details, <jobUrl>/executions/<executionId> (<executionUrl>);
• API to stop execution, <executionUrl>/stop;
• API to get logs(<executionUrl>/stdout), stdout (<executionUrl>/stdout) and stderr (<executionUrl>/stderr) for execution;
• Job execution log output streaming using WebSockets (ws://<executionUrl>/ws) and <executionUrl>/stream;
• Additional Job Metadata (useful for webhooks);
• Configurable retries per job;
• Configurable exponential back-off for retries and failures in monitoring of jobs;
• Configurable hard timeout for each execution;
• E-mail subscription to tasks;
• Healthcheck of the MongoDB and Redis Connections;
• Web hooks on job start;
• Web hooks on job completion;
• Redact any env that contains blacklisted keywords;
• Exponential back-off parameters per job;
• Self-healing handling of interrupted jobs;
• Self-healing handling of unscheduled periodical jobs;
• Workers should handle SIGTERM and exit gracefully;
• Docker⁠ Container Runner (with docker⁠ host pool);
• Docker⁠ Pool per task name (Regular Expressions);
• Rename docker⁠ containers after processing their details;
• Command to prune processed containers;
• Routes to remove/put back docker⁠ host in job balancing;
• Docker⁠ SSL connections;
• Circuit breaking when docker⁠ host is unavailable;
• Container Environment Variables per Job;
• Configurable global limit for number of running jobs per task name (Regular Expressions);
• Limit of concurrent job executions per task;
• Kubernetes Container Runner;
• MongoDB⁠ Task and Job Storage;
• Structured Logging;
• Monitoring of job completion;
• Job Expiration;
• Status Page with details on the farm status (executors, scheduled tasks and queue sizes);
• Error handling mechanism (Sentry built-in, extensible)
• Per-job Error handling mechanism (Sentry built-in, extensible)
• Usage metrics (extensible);
• Support Redis⁠ and Redis⁠ Sentinel;
• Support Redis⁠ Cluster;
• Comprehensive test coverage;
• CORS headers in every API request (configurable);
• Basic Auth support;
• gzip all JSON responses for the API (for requests that accept gzip);
• Store IP address of enqueued job for auditing (X-Real-IP, then X-Forwarded-For, then request.addr);
• Admin to inspect tasks and jobs.

⁠Getting started

Getting fastlane⁠ up and running is very simple if you have both docker⁠ and docker-compose⁠ installed.

We'll use a sample docker compose that gets all our requirements up (ps: this docker-compose⁠ file runs Docker In Docker⁠ and requires privileged mode to run):

$ curl https://raw.githubusercontent.com/fastlane-queue/fastlane/master/docker-compose-sample.yml | docker-compose -f - up -d

Starting fastlane...
Creating fastlane_mongo_1       ... done
Creating fastlane_docker-host_1 ... done
Creating fastlane_redis_1       ... done
Creating fastlane_fastlane_1    ... done
fastlane started successfully.

After that you can start using fastlane⁠. For more details on getting started, read the following page⁠.

⁠Documentation

Read more about fastlane at read the docs⁠.

⁠Contributing

Logo was created by Artur Sousa⁠.

⁠License

Fastlane is licensed under the MIT license⁠.