seanmorris/ids.aptcache repository overview

⁠SeanMorris/Ids v00.0.0

/ eye dee ess / • The PHP + Docker Framework

The Ids library provides general domain-primitives for developing web based applications. Routing, requests, modeling configuration, logging, sessions, and database access are all abstracted behind simple, expressive interfaces to efficient and powerful code.

To prevent unexpected behavior, the system is set to die on all errors down to E_NOTICE, excluding E_DEPRECATED errors generated from the vendor directory.

The project is made to run in docker but doesn't require it. It can be included in any composer project, and used in part or in whole easily.

The philosophy of the Ids project is headlined by security, speed and easy of use, in that order.

View the docs at docs.ids.seanmorr.is⁠

github.com/AlDanial/cloc v 1.81  T=0.13 s (1229.2 files/s, 199748.0 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
PHP                              86           3270            475          13720
JSON                             11              0              0           3909
Markdown                          2            928              0           1879
YAML                             25            118             14           1194
make                              2            145              9            610
SVG                              25              0              3            239
Bourne Shell                      4              9              0             40
Bourne Again Shell                2             17             17             23
HTML                              1              0              0             13
INI                               3              0              0             11
XML                               1              0              0              6
JavaScript                        1              0              0              1
CSS                               1              0              0              1
--------------------------------------------------------------------------------
SUM:                            164           4487            518          21646
--------------------------------------------------------------------------------

built by sean @ Tue 06 Apr 2021 04:52:47 AM EDT

⁠Installation

Include Ids in your project with:

$ composer require seanmorris/ids

Install Ids globally for access to the idilic cli tool:

$ composer global require seanmorris/ids:dev-master

Add composer's global vendor/bin to your PATH by adding this to your ~/.bashrc.

export PATH="$HOME/.composer/vendor/bin:$PATH"

⁠Creating a New Ids Project

Create a new project with composer, enter the directory and start php, apache & mysql:

$ composer create-project seanmorris/ids-project -s dev --remove-vcs
$ cd ids-project
$ make @dev start-bg

Thats it!

The companion package that provides a template for new projects can be found here:

https://github.com/seanmorris/ids-project⁠

⁠Dependencies

• Composer
• Docker
• Docker Compose
• GNU Make
• Git
• Linux or Compatible OS
• PHP
• SimpleTest
• Minikube (required for kubernetes targets only)

⁠Dev Tools

The dev build target provides facilities for connecting to xdebug and graylog.

⁠XDebug

XDebug is built into the dev images by default. You can configure it by setting XDEBUG_CONFIG_ environment variables in .env.dev. By default it will attempt to connect to port 9000 on ${DHOST_IP}, which is the machine runing the project.

⁠Apt Cache

Rebuild your images offline

Build the project with +aptcache with an internet connection to populate your cache.

$ make @dev+aptcache build

... to be continued

⁠GrayLog

\SeanMorris\Ids\Logger\Gelf provides a simple interface to graylog. Just add it to the IDS_LOGGERS_ environment variable to enable it. So long as there is a graylog server available , it will send all logs that meet the verbosity threshold.

The default graylog config for the dev target looks like:

IDS_LOGGERS_=\SeanMorris\Ids\Logger\Gelf
IDS_GRAYLOG_HOST=graylog
IDS_GRAYLOG_PORT=12201

This package comes with a default GELF TCP input in its graylog config backup. You can run make graylog-restore after starting graylog for the first time to create the input.

The graylog config can be backed up and restored with the following commands:

$ make graylog-backup     # alias glbak
$ make graylog-restore    # alias glres

Graylog can be started and stopped with the following commands:

$ make graylog-start      # alias gls
$ make graylog-start-fg   # alias glsf
$ make graylog-start-bg   # alias glsb

$ make graylog-stop       # alias gld

$ make graylog-restart    # alias glr
$ make graylog-restart-fg # alias glrf
$ make graylog-restart-bg # alias glrb

⁠Idilc CLI

Ids comes with the idilic command when installed globally with composer. When executed, it will ascend through the current path looking for another project including Ids. When it finds it, it will attach to that project and utilize its facilities.

If it encounters a local idilic pass-thru in the project directory, it will hand control to the docker environment where execution will continue.

Run idilic help to see actions exposed by any installed packages.

⁠Multistaging / Environment Targets

⁠base, prod, dev, and test

By default Ids provides 4 build targets: base, prod, dev, and test. Each exposes different ports, so they may run without conflict in parallel.

• base exposes no ports and builds without require-dev.

• prod exposes port 80 (configurable by IDS_EXPOSE_HTTP) and port 443* ( configurable by IDS_EXPOSE_HTTPS ) and builds without require-dev.

• test exposes port 2021 (configurable by IDS_EXPOSE_HTTP) and port 3031 ( IDS_EXPOSE_SQL ) and builds with require-dev.

• dev exposes port 2020 ( configurable by IDS_EXPOSE_HTTP ) and port 3030 ( IDS_EXPOSE_SQL ) and builds with require-dev.

*Not yet implemented.

⁠Switching Targets

The system will use the TARGET environment variable to decide which build target to use.

If youre on the BASH shell simply run one of the following commands to set the target for the context of a single command:

• make @base start
• make @dev build
• make @test test
• make @prod push-images

If you don't feek like typing @target for every command, you can ask the system to remember a target with:

• make stay@base
• make stay@dev
• make stay@test
• make stay@prod

⁠Extending Environments

Additional targets may be specified by creating a new infra/compose/[TARGET].yml in your project. Build this file as you'd build any other docker-compose file.

Then create a new .env.[TARGET] file in config/. Add your defaults here.

If you need some target specific build steps. then add a FROM base as TARGET section to any docker files in infra/docker that are relevant to that build target. You can expand on any existing build target, you're not limited to extending base.

⁠Build

The project may be built with make. Debian users can get this tool by running apt-get install build-essential. The build process also requires docker and docker-compose. minikube is required only for kubernetes testing.

The default build target is base. run make stay@dev or make stay@test to switch to the development/testing target. You can use @TARGET at the beginning of any make command to use the target for just that command.

$ make @dev build # build the project

$ make @dev start # start the services

Docker & docker-compose are available here:

• https://docs.docker.com/install/⁠
• https://docs.docker.com/compose/install/⁠

⁠Start / Stop / Restart

Make sure to set a target with make stay@target before issuing any commands. Alternatively you can run commands in the form make @target command.

$ make start      # Start the project in the background,
                  # with no output

$ make start-fg   # Start the project in the foreground.

$ make start-bg   # Start the project in the background,
                  # stream output to foreground.

$ make stop       # Stop all services defined for the target.

$ make stop-all   # Stop all services spawned for the target
                  # even ones no longer in target compose file.

$ make restart    # Stop, then restart the project in the background,
                  # with no output.

$ make restart-fg # Stop, then restart the project in the foreground.

$ make restart-bg # Stop, then restart the project in the background,
                  # stream output to foreground.

⁠Images & Tags

$ make list-images # List all images for the current project, target & branch.

$ make list-tags   # List all tags for the current project, target & branch.

$ make push-images # push all images for the current project, target & branch.

$ make pull-images # push all images for the current project, target & branch.

⁠Autotagging / Autopublishing / Git Hooks

Register git hooks with make hooks.

Image tags are generated automatically on build based on the date, current git tag (falls back to commit hash).

The master branch will generate:

• repository/project:gitTag-target
• repository/project:date-target
• repository/project:latest-target

Branches other than master will generate:

• repository/project:gitTag-target-branch
• repository/project:date-target-branch
• repository/project:latest-target-branch

Images will be built on git commit and pushed on git push if the current branch appears in the project root .publishing file in the form: BRANCH:TARGET. For example this file would push images for prod & test when git push is run for the master branch:

master:test
master:prod

⁠Available Images

Docker images for seanmorris/ids.idilic & seanmorris/ids.server for targets base, dev, and test are available for use & extension on DockerHub:

• idilic https://hub.docker.com/repository/docker/seanmorris/ids.idilic⁠
• server https://hub.docker.com/repository/docker/seanmorris/ids.server⁠

Pull from the cli with:

$ docker pull seanmorris/ids.idilic:latest
$ docker pull seanmorris/ids.server:latest

or extend in a dockerfile with one of the following:

(it is not recommended to use latest in FROM)

FROM seanmorris/ids.idilic:TAGNAME
FROM seanmorris/ids.server:TAGNAME

⁠Service Switches

Additional services can be included when running Ids. Simply add +toolname to your @target when issing a make command, and the necessary composer files will be included in the comtext of your command. For example, to start the system with graylog in dev mode:

make @dev+graylog start

Or inotify

make @dev+inotify start

Or both

make @dev+graylog+inotify start

⁠Templating

This ain't your granpappy's makefile.

GNU Make has a distinct and powerful syntax. It borrows, or in some instances outright uses the underlying bash engine to allow a user to define complex build steps and track the relationships between them to ensure everything is up to date.

Make allows for recursive variable expansion, conditionals, loops, and even calls out to the shell. This syntax is no longer limted to the Makefile.

Simply put the extension *.idstmp.* before your existing file extension. Ids will look for these files in all subdirectories of the prohect and rebuild them on startup. The resulting file will have the extension *.___gen.* where the *.idstmp.* is in the source file.

The resulting files should be excluded from version control, as they may contain artifacts from one target that should not exist in another.

For example: infra/docker/aptcache.idstmp.dockerfile starts off with the following line:

FROM ${BASELINUX}

Varibles are not normally allowed in the FROM section of dockerfile, preprocessed by make before it is used. So long as the file extenstion begins with .idstmp., we can count on a .___gen. file being produced. This allows us to keep all the images and containers synced to one base image.

The follwing lines from the end of the file show how one can use the shell to track who generated the file and when:

# generated @ $(shell date)
# by ${shell whoami} @ ${shell hostname}

See Functions for Transforming Text⁠ for more information.

⁠Configuration / Environment Variables / Secrets

⁠Loading Settings

Settings may be provided in environment variables, .env files, or yml files.

⁠Environment Variable & Target Files

The following files may be created/modified to configure the system. When the project is built, restarted, etc, they will be checked for modification, and re-built to the root of the projct if need be. The files generated to the root of the project should not be committed to version control.

• config/.env - Should not be committed to version control. Contains configuration that applies to the system regardles of the system's target.
• config/.env.default - Should be committed to version control. Contains non-secret configurations file and blank/dfault values for variables to be set in config/.env.
• config/.env_TARGET - Should not be committed to version control. Contains configuration based the system's target.
• config/.env_TARGET.default - Should be committed to version control. Contains non-secret configurations file and blank/dfault values for variables to be set in config/.env_TARGET.

The values set will be read according to the following precedence (higher takes precedence over lower):

• .env_TARGET
• .env_TARGET.default
• .env
• .env.default

Environment variables to be used in configuration should have the the prefix IDS_. An environment variable with the name IDS_SOME_VAR and IDS_SOME_OTHERVAR would be accessible within the system with:

IDS_SOME_VAR=value
IDS_SOME_OTHERVAR=other value

<?php
use \SeanMorris\Ids\Settings;

$someVar = Settings::read('some', 'var');
$someOtherVar = Settings::read('some', 'otherVar');

⁠Configuration Objects

They'd also both be accessible as an object, which can be iterated:

<?php

$some = Settings::read('some');

$some->var;
$some->otherVar;

foreach($some as $configKey => $value)
{

}

⁠Configuration Arrays

Arrays can be created by adding an underscore to the end of the environment variable name. The value will be split on whitespace.

IDS_ARRAY_=first second third

<?php
$array = Settings::read('ARRAY');

Whitespace will be preserved if the value is quotes:

IDS_ARRAY_=first "second element" third

Quotes can be escaped by doubling:

IDS_ARRAY_=first "second element ""with quotes inside""." third

⁠Hostname & Port based configuration

Hostname specific environment variables are prefixed with an extra underscore: IDS__. Dots and other non-word charaters in the hostname are placed by a single underscore, except for the hyhpen which is replaced by 3 underscores. Another double underscore finishes the hostname, and the variable name comes next.

The above environment variables could be overridden for example.com with: IDS__EXAMPLE_COM__SOME_VAR and IDS__EXAMPLE_COM__SOME_OTHERVAR. They would be accessed in the same way as above:

IDS__EXAMPLE_COM__SOME_VAR=overridden value
IDS_SOME_OTHERVAR=other overridden value

<?php
use \SeanMorris\Ids\Settings;

$someVar = Settings::read('some', 'var');
$someOtherVar = Settings::read('some', 'otherVar');

$some = Settings::read('some');

$some->var;
$some->otherVar;

Example.com could override the variables by port number if they wanted to change some behavior based on whether the user was on SSL. Adding another double underscore between the hostname and the variable name allows us to do that: IDS__EXAMPLE_COM__80__SOME_VAR and IDS__EXAMPLE_COM__80_SOME_OTHERVAR for HTTP and IDS__EXAMPLE_COM__443__SOME_VAR and IDS__EXAMPLE_COM__443_SOME_OTHERVAR for HTTPS.

Again, nothing changes in way they are accessed. The code reads them in the same way:

IDS__EXAMPLE_COM__80__SOME_VAR=normal value
IDS__EXAMPLE_COM__443__SOME_VAR=value for SSL

IDS__EXAMPLE_COM__443__SOME_VAR=some other value
IDS__EXAMPLE_COM__443_SOME_OTHERVAR=some other value for SSL

<?php
use \SeanMorris\Ids\Settings;

$someVar = Settings::read('some', 'var');

// ...and so on

⁠Non-Secret Config

Default values for environment variables that are non-secrets, (and thus may be pushed to version control) may be set in config/.env. Target specific variables may be set in config/.env.[TARGET]. DO NOT PUT SECRETS SUCH AS PASSWORDS HERE. These files will be checked into version control.

These files will be used to generate the final .env files in the root of the project when the project is built or started.

⁠Disposable Secrets

If you need a quick random value to use in the build process (ie when creating a MYSQL password), make sure to add the key to the relevant .env file, but leave the value blank. Add the key name to the .entropy file in the root of your project in the form: CONFIG_KEY:ENTROPY_KEY. CONFIG_KEY is the name of your configuration value, and ENTROPY_KEY is an arbitrary string that allows you to re-use the same random value between different configuration keys.

When the project starts, the .env files will be generated to the root of the project if they do not exist already, or if the existing ones are empty. Any keys present in the .entropy file will be set to a random 32 character string. This is how the schema name, username, and password are generated for the mysql server for the make test command when if the current TARGET is test.

run TARGET=test idilic info to see an example of this.

⁠Normal Secrets

So long as the .env files exist, and are not empty, the system will not attempt to regenerate them, so any secrets could in theory be added here, in place. This works fine for development, but for production it is recomended that secrets be stored in environtment variables.

Values from environment variables and .env files may accessd via PHP's getenv(). Ensure you've set the relevent values in the environment section of the target's docker-compose file if you chose not to add the value to a .env file.

⁠Json/Yaml Configuration

⁠Settings

Json/Yaml files may be provided in the config/ directory, under a directory named hostname/ or hostname;port/, where hostname and port are the domain and port you expect requests to come in on. The file should then be named settings.json or settings.yml. A directory named _;80 or ;80 will match only for requests on port 80 regardless of the hostname. A directory named _;, _, or ';' may be provided as a fallback if no others match.

NOTE If any yml config file is loaded, the system stops there and does not look for others. Host or port specific yml files will not be merged with more general wild card ones.

⁠Defaults

Default, non-secret values may be provided in a file named settings.defaults.yml or settings.defaults.json. Defaults are the same as settings except they're stored in version control.

Defaults are loaded by the same rules as standard settings files. Once a singe defaults file is found no more will be loaded.

Settings and Defaults files do not need to be loaded at the same level of generality. settings.yml may be found in _;80 and settings.defaults.yml may be found in hostname or vice versa.

Settings always take precedence over d