devopscorner/terraform-emr repository overview

⁠IaC Terraform EMR

DevOps Corner Indonesia - Terraform EMR Container Image (Provisioning Amazon EMR cluster using Terraform as Infrastructure-as-Code (IaC) tools)

---

⁠Available Tags

⁠Alpine

⁠Ubuntu

⁠AWS Linux (CodeBuild)

---

⁠Prerequirements

• Docker (docker)
• Docker Compose (docker-compose)
• AWS Cli version 2 (aws)
• Terraform Cli (terraform)
• Terraform Environment (tfenv)

⁠Build Container CI/CD

• Read full documentation, go to this⁠ link.

⁠Terraform EMR

• Clone this repository

  git clone https://github.com/devopscorner/iac-terraform-emr.git
  

  Copy

• Get Terraform Modules

  • Officials

    ./get-officials.sh
    -- or --
    make sub-officials
    

    Copy

  • Community

    ./get-community.sh
    -- or --
    make sub-community
    

    Copy

  • Get All Modules (Officials & Community)

    make sub-all
    

    Copy

• Provisioning your Infra (non existing infrastructure)

  • Goto terraform/environment/providers/aws/infra

    cd core
    terraform init
    terraform workspace select staging
    terraform plan
    terraform apply
    

    Copy

• Provisioning your Terraform State (Remote State)

  • Goto terraform/environment/providers/aws/tfstate

    cd core
    terraform init
    terraform workspace select staging
    terraform plan
    terraform apply
    

    Copy

• Provisioning Amazon EMR (existing infrastructure)

  • Goto terraform/environment/providers/aws/infra/resources

    cd emr
    terraform init
    terraform workspace select staging
    terraform plan
    terraform apply
    ---
    notes: you need to define your existing infra in variables.tf
    

    Copy

• Provisioning Other Infra Resources

  • Read provisioning other infra resources, go to this⁠ link.

⁠Terraform EMR Inside Container

• ECR Build

  • Example:

    # Alpine
    
    docker build . -t YOUR_AWS_ACCOUNT.dkr.ecr.ap-southeast-1.amazonaws.com/devopscorner/terraform-emr:alpine
    

    Copy

  • With Script:

    make build-tf-emr ARGS=YOUR_AWS_ACCOUNT CI_PATH=devopscorner/terraform-emr
    

    Copy

• ECR Tag

  • Example:

    # Alpine
    
    docker tag YOUR_AWS_ACCOUNT.dkr.ecr.ap-southeast-1.amazonaws.com/devopscorner/terraform-emr:alpine YOUR_AWS_ACCOUNT.dkr.ecr.ap-southeast-1.amazonaws.com/devopscorner/terraform-emr:latest
    

    Copy

  • With Script:

    make tag-tf-emr ARGS=YOUR_AWS_ACCOUNT CI_PATH=devopscorner/terraform-emr
    

    Copy

• ECR Push

  • Example:

    # Alpine
    
    docker push YOUR_AWS_ACCOUNT.dkr.ecr.ap-southeast-1.amazonaws.com/devopscorner/terraform-emr:alpine
    
    # Latest (Alpine)
    
    docker push YOUR_AWS_ACCOUNT.dkr.ecr.ap-southeast-1.amazonaws.com/devopscorner/terraform-emr:latest
    

    Copy

  • With Script:

    make push-tf-emr ARGS=YOUR_AWS_ACCOUNT CI_PATH=devopscorner/terraform-emr
    

    Copy

⁠Using Cloud9 IDE

• Provisioning Cloud9 in EC2 onspot Instances

  aws cloud9 create-environment-ec2 --name example-env --description "environment" --instance-type t3.small --subnet-id subnet-id --automatic-stop-time-minutes 60 --owner-arn arn:aws:iam::123:user/User
  
  -- or --
  
  Go to `terraform/environment/providers/aws/infra/resources/cloud9` folder
  
  terraform init
  terraform workspace select staging
  terraform plan
  terraform apply
  

  Copy

⁠Cleanup

• Destroy Environment Lab

  terraform destroy
  

  Copy

⁠Tested Environment

⁠Versioning

• Docker version

  docker -v
  ---
  Docker version 20.10.17-rd, build c2e4e01
  
  docker version
  ---
  Client:
    Version:           20.10.17-rd
    API version:       1.41
    Go version:        go1.17.11
    Git commit:        c2e4e01
    Built:             Fri Jul 22 18:31:17 2022
    OS/Arch:           darwin/amd64
    Context:           default
    Experimental:      true
  
  Server: Docker Desktop 4.14.1 (91661)
  Engine:
    Version:          20.10.21
    API version:      1.41 (minimum version 1.12)
    Go version:       go9.7
    Git commit:       3056208
    Built:            Tue Oct 25 18:00:19 2022
    OS/Arch:          linux/amd64
    Experimental:     false
  containerd:
    Version:          1.6.9
    GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
  runc:
    Version:          1.1.4
    GitCommit:        v1.1.4-0-g5fd4c4d
  docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  

  Copy

• Docker-Compose version

  docker-compose -v
  ---
  Docker Compose version v2.11.1
  

  Copy

• AWS Cli

  aws --version
  ---
  aws-cli/2.8.7 Python/3.9.11 Darwin/21.6.0 exe/x86_64 prompt/off
  

  Copy

• Terraform Cli

  terraform version
  ---
  Terraform v1.3.5
  on darwin_amd64
  - provider registry.terraform.io/hashicorp/aws v3.74.3
  - provider registry.terraform.io/hashicorp/local v2.1.0
  - provider registry.terraform.io/hashicorp/null v3.1.0
  - provider registry.terraform.io/hashicorp/random v3.1.0
  - provider registry.terraform.io/hashicorp/time v0.7.2
  

  Copy

• Terraform Environment Cli

  tfenv -v
  ---
  tfenv 2.2.2
  

  Copy

⁠Security Check

Make sure that you didn't push sensitive information in this repository

• AWS Credentials (AWS_ACCESS_KEY, AWS_SECRET_KEY)
• AWS Account ID
• AWS Resources ARN
• Username & Password
• Private (id_rsa) & Public Key (id_rsa.pub)
• DNS Zone ID
• APP & API Key

---

⁠Repository URL

• DevOpsCorner Terraform EMR⁠

---

⁠Copyright

• Author: Dwi Fahni Denni (@zeroc0d3)
• Vendor: DevOps Corner Indonesia (devopscorner.id)
• License: Apache v2