briteskies/local-docker repository overview

⁠local-docker

⁠Purpose:

A single repository for maintaining local Magento development environments using docker.

⁠Goals:

• Add as few files as possible/useful to the git repositories of projects. [Current count: 2]
• Reuse as much existing code and as many existing images as possible.
• Use Alpine Linux and "slim" versions of images to reduce image size where prudent.
• Include useful scripts to simplify building and maintaining a local environment.
• Keep the actual implementation completely customizable by the user, so new tools or software can be easily added and tested as required.
• Prevent surprise upgrades by storing Magento version specific images on dockerhub.

⁠Creating a New Magento 2 Cloud Project

⁠Reference the devdocs and your onboarding guide heavily for updates

⁠https://devdocs.magento.com/guides/v2.3/cloud/bk-cloud.html⁠

⁠https://cloud.magento.com/project/PROJECT_ID/dashboard/gettingstarted⁠

1. Get added to the project by the Magento contact, once server setup is complete.

2. Login to https://magento.com⁠ and navigate to https://account.magento.com/cloud/project/subscriptions/⁠, then click on the project.

3. Navigate to the Infrastructure Access and open the Project Access (Web UI).

4. Clone the project from Magento Cloud to your local environment and setup in phpstorm, making sure to switch to the integration branch.

5. Create the auth.json file from the env:COMPOSER_AUTH variable under the settings menu.

6. Add the following under repositories in the composer.json.

    "briteskies-local-docker": {
        "type": "vcs",
        "url": "[email protected]:briteskies/local-docker.git"
    },
    "briteskies-magento-pipelines": {
        "type": "vcs",
        "url": "[email protected]:briteskies/magento-pipelines.git"
    },
    "artifacts": {
        "type": "path",
        "url": "artifacts/*/*",
        "options": {
            "symlink": false
        }
    }
   

   Copy

7. Run composer require "briteskies/local-docker:dev-2_3 as 2.3.0" "briteskies/magento-pipelines:dev-2_3 as 2.3.0" --ignore-platform-reqs

8. Add the following require-dev and scripts

        "scripts": {
            "codestyle-fixer": [
                "phpcbf --standard=psr2 --ignore=*.css,*.js --warning-severity=8 app/code/",
                "phpcbf --standard=EcgM2 --ignore=*.css,*.js --runtime-set installed_paths vendor/magento-ecg/coding-standard --warning-severity=8 app/code/"
            ],
            "codestyle-check": [
                "phpcs --standard=psr2 --ignore=*.css,*.js --warning-severity=8 app/code/",
                "phpcs --standard=EcgM2 --ignore=*.css,*.js --runtime-set installed_paths vendor/magento-ecg/coding-standard --warning-severity=8 app/code/"
            ],
            "codestyle-check-warnings": [
                "phpcs --standard=psr2 --ignore=*.css,*.js --runtime-set ignore_warnings_on_exit true app/code/",
                "phpcs --standard=EcgM2 --ignore=*.css,*.js --runtime-set installed_paths vendor/magento-ecg/coding-standard --runtime-set ignore_warnings_on_exit true app/code/"
            ],
            "vulnerabilities-check": [
                "security-checker security:check"
            ]
        },
        "require-dev": {
            "cweagans/composer-patches": "^1.6",
            "lusitanian/oauth": "~0.8.10",
            "magento-ecg/coding-standard": "^3.1",
            "pdepend/pdepend": "2.5.2",
            "phpmd/phpmd": "^2.6",
            "phpunit/phpunit": "~6.2.0",
            "sebastian/phpcpd": "~3.0.0",
            "squizlabs/php_codesniffer": "3.2.2",
            "friendsofphp/php-cs-fixer": "~2.10.1",
            "briteskies/local-docker": "^1.0"
        }
   

   Copy

9. Run composer update

10. Add any urls required (including local and integration) to magento-var.php.

11. Adjust the settings of .magento.env.yaml

    • relationships add elasticsearch: "elasticsearch:elasticsearch"
    • web > locations > rules should be updated from |svgz| to |svgz?|
    • disk = 4096
    • Add any crons, environmental variables, or tweaks required.

12. Add the following rules to the magento/routes.yaml with redirects for each production url without a www⁠.

     "https://{default}/":
         type: upstream
         upstream: "mymagento:php"
    
     "http://*.{default}/":
         type: upstream
         upstream: "mymagento:php"
    
     "https://*.{default}/":
         type: upstream
         upstream: "mymagento:php"
    
    
     https://DOMAIN.com/:
         type: redirect
         redirects:
             expires: -1s
             paths: {}
         tls:
             strict_transport_security:
                 enabled: null
                 include_subdomains: null
                 preload: null
             min_version: null
             client_authentication: null
             client_certificate_authorities: []
         to: https://www.DOMAIN.com/
    

    Copy

13. Add the following to magento/services.yaml

     elasticsearch:
            type: elasticsearch:5.2
            disk: 1024
            configuration:
                plugins:
                  - analysis-icu
                  - analysis-phonetic
    

    Copy

14. Create an empty frontend theme for the project based off of Magento/blank. EG Cvi/default

15. Create the following files for grunt and force add them to the git repo: package.json, grunt-config.json, Gruntfile.js, dev/tools/grunt/configs/local-themes.js

16. Copy examples/docker-compose.example.yml and examples/docker-compose.windows-example.yml the root of the project and force add them into git.

17. Use one of the example files to setup your docker-compose.yml and customize the volumes as needed.

18. Copy examples/env.php.docker to app/etc/examples/env.php.docker and examples/env.php and force app/etc/examples/env.php.docker into git.

19. Run bash .docker/fresh-build.sh to create an empty site.

20. Verify that the site runs and commit changes back to the repository.

21. See the magento-pipelines project for setup of Bitbucket, Servers.

⁠Creating a New Magento 2 EE Project

1. Login to bitbucket.

2. Create a new repository

   • Select the "Magento 2.x Implementations" Project (eg. https://bitbucket.org/account/user/briteskies/projects/MAGE2⁠)
   • Name the project something short descriptive and url friendly.
   • Set it to be private.
   • Select to include a README file.
   • Select git as the VCS.
   • Open the Advanced Setting
     • Select "Only Private Forks"
     • Select Language as "PHP"

3. Clone the repository locally.

4. Edit the README.md to contain something along the lines of the following where [project-slug] is the url path for the project.

    # [project-slug]
    ###### A repository for the [project-slug] magento site.
   
   
    ### Getting Started
   
    - Follow the instructions in https://bitbucket.org/briteskies/local-docker/src/2_3/ to setup a local docker environment.
    - Follow the instructions under "New Development/Deployment Process Overview" https://bitbucket.org/briteskies/magento-pipelines/src/2_3/ to learn how to work with this project.
    - If bugs arise, search through this jira board for any known solutions https://briteskies.atlassian.net/projects/MAG/board/.
   
   
    ### Please document any peculiarities or recommendations for working the project here.
   
    - Update the above lines accordingly when the Magento Version changes.
    
   

   Copy

5. Copy examples/.gitignore.example into the root of the project as .gitignore.

6. Copy examples/composer.json into the root of the project.

7. Copy examples/docker-compose.yml into the root of the project as both docker-compose.yml and docker-compose.yml.example.\

8. Copy examples/env.php.docker to app/etc/examples/env.php.docker and examples/env.php.

9. Apply any customizations required to the docker-compose.yml.

10. Run composer install.

11. Run the following command on the host for magento 2.3 changing the root dir as needed. php dev/tools/UpgradeScripts/pre_composer_update_2.3.php --root ~/html/magento2-ee-2.3

12. Run composer update --ignore-platform-reqs.

13. Run bash .docker/docker-restart.sh.

14. Create a .docker/sql/scrub-db-variables.sql file from .docker/sql/scrub-db-variables.default.sql and customize it as needed.

15. Run bash .docker/fresh-build.sh.

16. Verify that the site runs and commit changes back to the repository.

⁠Adding Docker to a Project

1. Set the minimum-stability to beta in the composer.json of the project.

2. Add the following to the composer.json of the project under the repositories section. The key is optional depending on the file.

    "briteskies-local-docker": {
      "type": "vcs",
      "url": "[email protected]:briteskies/local-docker.git"
    },
   

   Copy

3. Run composer require briteskies/local-docker:dev-2_3 as 2.3.0 --ignore-platform-reqs.

   • Rerun this command to pull in any updates to the environment and update the composer.lock.
   • Custom branches can be created if needed, but should be avoided as much as possible.
   • NOTE: composer doesn't seem to allow for periods in branch names.

4. Copy .docker/examples/env.php to app/etc/env.php.docker and apply and customizations that are desired for the project.

    cp .docker/examples/env.php.docker app/etc/env.php.docker
   

   Copy

5. Copy .docker/examples/docker-compose.yml to docker-compose.yml.example and apply and customizations that are desired for the project.

    cp .docker/examples/docker-compose.yml docker-compose.yml.example
   

   Copy

6. Add the following to the .gitignore file for the project or use the .docker/examples/.gitignore

    # Docker
    .docker
    docker-compose.yml
   

   Copy

7. Commit and push the updates to back to the project repository.

⁠Local Environment Setup:

1. Follow docker setup instructions for your OS in the docs/SETUP-DOCKER.md
2. Download the project's git repository.
3. cd to your project directory.
4. Run composer install --ignore-platform-reqs.
5. Copy app/etc/env.php.docker to app/etc/env.php
   • Adjust any settings to your liking.
6. Copy .docker-compose.yml.example to docker-compose.yml.
   • Go through each of the volumes in the file and adjust the host paths (the part before the :)to match the actual file and folder location on your system.
     • To make your own files that are safe from being overwritten copy the file and prepend custom. to the beginning of the file.
   • Adjust any settings or to your liking.
   • Remove unwanted services by commenting them out.
   • Completely customize a service by commenting out the image and adding a build field.
7. Run sudo chmod +x .docker/*.sh to fix file permissions.
8. Run bash .docker/sync-with-vendor.sh or bash .docker/sync-with-artifacts.sh to copy files that magento misses.
9. Start docker with bash .docker/docker-restart.sh or docker-compose up -d.
   • NOTE: This currently throws an http error after the containers have been started, but doesn't seem to impact the environment.
   • If you mapping ports for a specific service, you'll need to stop using those ports on the host machine or adjust the ports in the docker-compose.yml (the part before the :) to ones that are available on your host.
10. (Optional) Copy .docker/sql/scrub-db-variables.default.sql to .docker/sql/scrub-db-variables.sql and customize any variables to your liking.
11. (Optional) Download a gzipped copy of the database to .docker/sql/backups and run bash .docker/refresh-database.sh
    • Otherwise run bash .docker/fresh-build.sh or bash .docker/install-database.sh (TODO for Setup Performance and Demo sites.)
    • Otherwise run bash .docker/php-connect.sh and run your own commands to install the database.
12. Add 127.0.0.1 local.briteskies.com to your hosts file, or add an entry for the new url if you changed the site url in the above step.
13. Run .docker/fresh-install.sh (~15m)
    • For subsequent updates run bash .docker/deploy-site.sh (~6m), .docker/deploy-site-fast.sh (~6m - ~30s), bash docker/refresh-database.sh (~6m), or bash docker/reindex.sh (~6m) as needed.
    • times based on Intel i7-6700HQ (8) @ 3.500GHz, 16GB Ram, and a sata SSD, on the SMKWM2 project.
14. Stop docker with docker-compose down
15. To access the magento command line run bash .docker/php-connect.sh

⁠xDebug Setup

• TODO
• Try toggling xdebug.remote_connect_back=1 in .docker/contexts/apache/httpd-vhosts.conf
• Ensure the server name here .docker/contexts/apache/httpd-vhosts.conf:20 mataches both the name and host of the server in phpstorm.

⁠Useful urls (Depends on the specific docker-compose.yml)

⁠Docker Creation Overview - (Not required to use Docker with a project):

1. Download this repo.

   • If downloaded a part of a project, this environment can be edited directly by copying the .git folder from the vendor directory

     cp -a vendor/briteskies/local-docker/ .docker
     

     Copy

2. Test building from the docker-compose.yml:

    # starting the from this repositories root.
    cd examples
    
    # build docker environment (requires symlinks to be working)
    docker-compose down ; docker-compose up --build --remove-orphans --verbose
    
    # view http://localhost:8080/info.php in a browser to confirm
    # add any additional sites or data for testing to `examples/pub`
   

   Copy

3. Edit, Commit, and Push changes branching accordingly (eg 1_14, 2_0, 2_1, 2_3, etc).

4. hub.docker.com is automatically triggered to build/rebuild the docker image. (TODO for Setup Performance)

5. After build completes download and use. (TODO for Setup Performance)

⁠Docker Maintenance Steps:

1. Follow the steps in Docker Creation Overview or cd .docker in an existing project and use it like any other git repo.
2. Run composer require briteskies/local-docker:dev-X_X --ignore-platform-reqs where X_X are the first two Magento version numbers.
3. Make any recommended changes to the docker-compose.yml.example and app/etc/env.php.docker.
4. Commit and push the changes back to the project.

⁠local-docker Project Structure

• contexts - The build context of the various Dockerfile's needed for Magento
  • (all) - folders are named to match the service in the docker-compose.yml that they're conditionally built from.
• docs
  • SETUP-DOCKER.md - Instructions for installing docker by OS.
  • SETUP-DEVELOPER.md - Instructions for installing software for development and debugging leaning towards platform independent tools. (TODO for Knowledge sharing and onboarding)
• examples - Default files that can be used for installing/configuring a site or environments.
  • .docker - A recursive symlink to this repository's root for easy testing of the docker-compose.yml
  • pub - A folder to mimic magento's pub folder when testing without magento.
  • .gitignore - A full example .gitigore with comments file that can be copied to a project
  • composer.json - A full example composer.json for starting a new project.
  • docker-compose.yml - An example docker-compose.yml for the Magento version that's currently checked out.
  • env.php.docker - An example env.php with docker service names inserted as the hosts.
• hooks - Git hooks that can be (optionally) copied to .git/hooks in the project's repository.
  • pre-commit - Runs php code sniffer before allowing a commit.
• media - A location to store pub/media archives for local use.
• sql - A location to store miscellaneous sql scripts for db scrubbing or performing specific tasks.
  • backups - A location to store database dumps. The latest file will be used when refreshing the database.
  • scrub-db-variables.default.sql - Default sql variables for the scrub db script.
  • scrub-db-variables.sql - A git ignored file for overriding one or more variables.
  • scrub-db-z.sql - A script for scrubbing the database of production sensitive info and setting it up for local development. Named "z" to ensure that concatenating the files named scrub-db-* will place it at the end.
• .git-ignore - For ignoring files or holding open empty directories.
• composer.json - For adding this repository to projects via composer.
• docker-restart.sh - Start/restart docker with a few additional arguements.
• deploy-site.sh - Redeploy everything from scratch.
• deploy-site-fast.sh - Redeploy using the quickest methods possible.
• fresh-build.sh - Triggers docker/refresh-database.sh, .docker/deploy-site.sh, anddocker/reindex.sh repectively.
• Makefile - (TODO for a potential way to simplify the commmands installed)
• php-connect - Enter the php contain in the command line to run magento commands. (mage and magerun are available aliases)
• README.md - You are here.
• refresh-database.sh - Install and scrub the newest dump from sql/backups.
• reindex.sh - Shortcut to reindex everything.
• reindex-fast.sh - (TODO for setup Performance)
• toggle-cron.sh - (TODO for Dev convenience)
• toggle-xdebug.sh - (TODO for Dev convenience (could help noticeably with local performance))

⁠Ideas/Notes/TODO

• Add a script to remove all images and containers by project.
  • https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes⁠
• Add add a script to remove all storage volumes for a project.
• remove extra settings from redis env.php.docker
• Move shell script variables into a single file include the .default. file then include the one without default like the scrub-db-variables.default.sql
  • https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md⁠
• Add another nginx instance in front of varnish for ssl termination using a Self Signed Cert
• Separate redis session and cache instances.
• Add location to download project db's by parent folder names.
  • Add an env file to set any sensitive credentials along with and example file.
• For cli commands see: https://docs.docker.com/engine/reference/commandline/docker/⁠
• Use git branch aliases to enable tracking of precise releases of magento without adding weight. EG (2_3_1, 2_3_2, 2_3_3, and 2_3_4 point to 2_3 unless they have special occurences,)
  • only need to trigger the unique ones in prod.
  • delete master branch or
• Provide clear and concise instructions such that anyone can quickly setup a local environment.
  • Eg. Managers, QA's, and Clients.
• Include useful git hooks, to automatically update local environments when pulling or checking out a branch.
  • Make a git pull hook that alerts the developer when docker-compose.yml.example has been updated.
• Add script to download fresh databases.
• Turn these instructions into one or more blog posts.
• find images / plugins for hosts management, file sync, and logging
• learn how to use the makefile for simplifying interactions with the site, or other bash auto-completion
• Fix container user permissions.
• Fix xdebug setup.
• Lint JS and LESS Q4 2018
• Blackfire setup.

⁠General Notes on Building Docker Images

• Building a docker image is separated into steps based on each command in the Dockerfile.
• Each step is cached by a hash.
• When building, if the hash of the step is different, the cache is discard and each subsequent step is rebuilt.
  • Meaning a change added to the end of a Dockerfile is much faster to rebuild.
  • Also changes made to files in the context will not be picked up automatically, so a full rebuild must be forced.
• Use of the && from shell scripting is used prolifically to reduce build steps.
• To reduce image size, remove any extraneous files that were generated at the end of each step.
• Images can also be created, by committing a running container, but a defining everything in the Dockerfile is still preferred.

⁠Glossary