bridgecrew/terraformer repository overview
โ Terraformer
A CLI tool that generates tf/json and tfstate files based on existing infrastructure
(reverse Terraform).
- Disclaimer: This is not an official Google product
- Created by: Waze SRE
โ Table of Contents
- Capabilitiesโ
- Installationโ
- Supported Providersโ
- Major Cloud
- Cloud
- Infrastructure Software
- Network
- VCS
- Monitoring & System Management
- Community
- Contributingโ
- Developingโ
- Infrastructureโ
โ Capabilities
- Generate
tf/json+tfstatefiles from existing infrastructure for all supported objects by resource. - Remote state can be uploaded to a GCS bucket.
- Connect between resources with
terraform_remote_state(local and bucket). - Save
tf/jsonfiles using a custom folder tree pattern. - Import by resource name and type.
- Support terraform 0.13 (for terraform 0.11 use v0.7.9).
Terraformer uses Terraform providers and is designed to easily support newly added resources. To upgrade resources with new fields, all you need to do is upgrade the relevant Terraform providers.
Import current state to Terraform configuration from a provider
Usage:
import [provider] [flags]
import [provider] [command]
Available Commands:
list List supported resources for a provider
Flags:
-b, --bucket string gs://terraform-state
-c, --connect (default true)
-ะก, --compact (default false)
-x, --excludes strings firewalls,networks
-f, --filter strings compute_firewall=id1:id2:id4
-h, --help help for google
-O, --output string output format hcl or json (default "hcl")
-o, --path-output string (default "generated")
-p, --path-pattern string {output}/{provider}/ (default "{output}/{provider}/{service}/")
--projects strings
-z, --regions strings europe-west1, (default [global])
-r, --resources strings firewall,networks or * for all services
-s, --state string local or bucket (default "local")
-v, --verbose verbose mode
-n, --retry-number number of retries to perform if refresh fails
-m, --retry-sleep-ms time in ms to sleep between retries
Use " import [provider] [command] --help" for more information about a command.
โ Permissions
The tool requires read-only permissions to list service resources.
โ Resources
You can use --resources parameter to tell resources from what service you want to import.
To import resources from all services, use --resources="*" . If you want to exclude certain services, you can combine the parameter with --excludes to exclude resources from services you don't want to import e.g. --resources="*" --excludes="iam".
โ Filtering
Filters are a way to choose which resources terraformer imports. It's possible to filter resources by its identifiers or attributes. Multiple filtering values are separated by :. If an identifier contains this symbol, value should be wrapped in ' e.g. --filter=resource=id1:'project:dataset_id'. Identifier based filters will be executed before Terraformer will try to refresh remote state.
โ Resource ID
Filtering is based on Terraform resource ID patterns. To find valid ID patterns for your resource, check the import part of the Terraform documentationโ .
Example usage:
terraformer import aws --resources=vpc,subnet --filter=vpc=myvpcid --regions=eu-west-1
Will only import the vpc with id myvpcid. This form of filters can help when it's necessary to select resources by its identifiers.
โ Planning
The plan command generates a planfile that contains all the resources set to be imported. By modifying the planfile before running the import command, you can rename or filter the resources you'd like to import.
The rest of subcommands and parameters are identical to the import command.
$ terraformer plan google --resources=networks,firewall --projects=my-project --regions=europe-west1-d
(snip)
Saving planfile to generated/google/my-project/terraformer/plan.json
After reviewing/customizing the planfile, begin the import by running import plan.
$ terraformer import plan generated/google/my-project/terraformer/plan.json
โ Resource structure
Terraformer by default separates each resource into a file, which is put into a given service directory.
The default path for resource files is {output}/{provider}/{service}/{resource}.tf and can vary for each provider.
It's possible to adjust the generated structure by:
- Using
--compactparameter to group resource files within a single service into oneresources.tffile - Adjusting the
--path-patternparameter and passing e.g.--path-pattern {output}/{provider}/to generate resources for all services in one directory
It's possible to combine --compact --path-pattern parameters together.
โ Installation
From source:
- Run
git clone <terraformer repo> - Run
go mod download - Run
go build -vfor all providers OR build with one providergo run build/main.go {google,aws,azure,kubernetes and etc} - Run
terraform initagainst anversions.tffile to install the plugins required for your platform. For example, if you need plugins for the google provider,versions.tfshould contain:
terraform {
required_providers {
google = {
source = "hashicorp/google"
}
}
required_version = ">= 0.13"
}
Or alternatively
- Copy your Terraform provider's plugin(s) to folder
~/.terraform.d/plugins/{darwin,linux}_amd64/, as appropriate.
From Releases:
- Linux
export PROVIDER={all,google,aws,kubernetes}
curl -LO https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-linux-amd64
chmod +x terraformer-${PROVIDER}-linux-amd64
sudo mv terraformer-${PROVIDER}-linux-amd64 /usr/local/bin/terraformer
- MacOS
export PROVIDER={all,google,aws,kubernetes}
curl -LO https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-darwin-amd64
chmod +x terraformer-${PROVIDER}-darwin-amd64
sudo mv terraformer-${PROVIDER}-darwin-amd64 /usr/local/bin/terraformer
โ Using a package manager
If you want to use a package manager:
- Homebrewโ users can use
brew install terraformer. - Chocolateyโ users can use
choco install terraformer.
Links to download Terraform Providers:
- Major Cloud
- Cloud
- Infrastructure Software
- Network
- Cloudflare provider >1.16 - hereโ
- VCS
- GitHub provider >=2.2.1 - hereโ
- Monitoring & System Management
- Community
Information on provider plugins: https://www.terraform.io/docs/configuration/providers.htmlโ
โ Use with GCP
Example:
terraformer import google --resources=gcs,forwardingRules,httpHealthChecks --connect=true --regions=europe-west1,europe-west4 --projects=aaa,fff
terraformer import google --resources=gcs,forwardingRules,httpHealthChecks --filter=compute_firewall=rule1:rule2:rule3 --regions=europe-west1 --projects=aaa,fff
For google-beta provider:
terraformer import google --resources=gcs,forwardingRules,httpHealthChecks --regions=europe-west4 --projects=aaa --provider-type beta
List of supported GCP services:
addressesgoogle_compute_address
autoscalersgoogle_compute_autoscaler
backendBucketsgoogle_compute_backend_bucket
backendServicesgoogle_compute_backend_service
bigQuerygoogle_bigquery_datasetgoogle_bigquery_table
cloudFunctionsgoogle_cloudfunctions_function
cloudsqlgoogle_sql_database_instancegoogle_sql_database
dataProcgoogle_dataproc_cluster
disksgoogle_compute_disk
externalVpnGatewaysgoogle_compute_external_vpn_gateway
dnsgoogle_dns_managed_zonegoogle_dns_record_set
firewallgoogle_compute_firewall
forwardingRulesgoogle_compute_forwarding_rule
gcsgoogle_storage_bucketgoogle_storage_bucket_aclgoogle_storage_default_object_aclgoogle_storage_bucket_iam_bindinggoogle_storage_bucket_iam_membergoogle_storage_bucket_iam_policygoogle_storage_notification
gkegoogle_container_clustergoogle_container_node_pool
globalAddressesgoogle_compute_global_address
globalForwardingRulesgoogle_compute_global_forwarding_rule
healthChecksgoogle_compute_health_check
httpHealthChecksgoogle_compute_http_health_check
httpsHealthChecksgoogle_compute_https_health_check
iamgoogle_project_iam_custom_rolegoogle_project_iam_membergoogle_service_account
imagesgoogle_compute_image
instanceGroupManagersgoogle_compute_instance_group_manager
instanceGroupsgoogle_compute_instance_group
instanceTemplatesgoogle_compute_instance_template
instancesgoogle_compute_instance
interconnectAttachmentsgoogle_compute_interconnect_attachment
kmsgoogle_kms_key_ringgoogle_kms_crypto_key
logginggoogle_logging_metric
memoryStoregoogle_redis_instance
monitoringgoogle_monitoring_alert_policygoogle_monitoring_groupgoogle_monitoring_notification_channelgoogle_monitoring_uptime_check_config
networksgoogle_compute_network
packetMirroringsgoogle_compute_packet_mirroring
nodeGroupsgoogle_compute_node_group
nodeTemplatesgoogle_compute_node_template
projectgoogle_project
pubsubgoogle_pubsub_subscriptiongoogle_pubsub_topic
regionAutoscalersgoogle_compute_region_autoscaler
regionBackendServicesgoogle_compute_region_backend_service
regionDisksgoogle_compute_region_disk
regionHealthChecksgoogle_compute_region_health_check
regionInstanceGroupsgoogle_compute_region_instance_group
regionSslCertificatesgoogle_compute_region_ssl_certificate
regionTargetHttpProxiesgoogle_compute_region_target_http_proxy
regionTargetHttpsProxiesgoogle_compute_region_target_https_proxy
regionUrlMapsgoogle_compute_region_url_map
reservationsgoogle_compute_reservation
resourcePoliciesgoogle_compute_resource_policy
regionInstanceGroupManagersgoogle_compute_region_instance_group_manager
routersgoogle_compute_router
routesgoogle_compute_route
schedulerJobsgoogle_cloud_scheduler_job
securityPoliciesgoogle_compute_security_policy
sslCertificatesgoogle_compute_managed_ssl_certificate
sslPoliciesgoogle_compute_ssl_policy
subnetworksgoogle_compute_subnetwork
targetHttpProxiesgoogle_compute_target_http_proxy
targetHttpsProxiesgoogle_compute_target_https_proxy
targetInstancesgoogle_compute_target_instance
targetPoolsgoogle_compute_target_pool
targetSslProxiesgoogle_compute_target_ssl_proxy
targetTcpProxiesgoogle_compute_target_tcp_proxy
targetVpnGatewaysgoogle_compute_vpn_gateway
urlMapsgoogle_compute_url_map
vpnTunnelsgoogle_compute_vpn_tunnel
Your tf and tfstate files are written by default to
generated/gcp/zone/service.
โ Use with AWS
Example:
terraformer import aws --resources=vpc,subnet --connect=true --regions=eu-west-1 --profile=prod
terraformer import aws --resources=vpc,subnet --filter=vpc=vpc_id1:vpc_id2:vpc_id3 --regions=eu-west-1
โ Profiles support
AWS configuration including environmental variables, shared credentials file (~/.aws/credentials), and shared config file (~/.aws/config) will be loaded by the tool by default. To use a specific profile, you can use the following command:
terraformer import aws --resources=vpc,subnet --regions=eu-west-1 --profile=prod
You can also provide no regions when importing resources:
terraformer import aws --resources=cloudfront --profile=prod
In that case terraformer will not know with which region resources are associated with and will not assume any region. That scenario is useful in case of global resources (e.g. CloudFront distributions or Route 53 records) and when region is passed implicitly through environmental variables or metadata service.
โ Supported services
accessanalyzeraws_accessanalyzer_analyzer
acmaws_acm_certificate
alb(supports ALB and NLB)aws_lbaws_lb_listeneraws_lb_listener_ruleaws_lb_listener_certificateaws_lb_target_groupaws_lb_target_group_attachment
api_gatewayaws_api_gateway_authorizeraws_api_gateway_documentation_partaws_api_gateway_gateway_responseaws_api_gateway_integrationaws_api_gateway_integration_responseaws_api_gateway_methodaws_api_gateway_method_responseaws_api_gateway_modelaws_api_gateway_resourceaws_api_gateway_rest_apiaws_api_gateway_stageaws_api_gateway_usage_planaws_api_gateway_vpc_link
appsyncaws_appsync_graphql_api
auto_scalingaws_autoscaling_groupaws_launch_configurationaws_launch_template
budgetsaws_budgets_budget
cloud9aws_cloud9_environment_ec2
cloudfrontaws_cloudfront_distribution
cloudformationaws_cloudformation_stackaws_cloudformation_stack_setaws_cloudformation_stack_set_instance
cloudtrailaws_cloudtrail
cloudwatchaws_cloudwatch_dashboardaws_cloudwatch_event_ruleaws_cloudwatch_event_targetaws_cloudwatch_metric_alarm
codebuildaws_codebuild_project
codecommitaws_codecommit_repository
codedeployaws_codedeploy_app
codepipelineaws_codepipelineaws_codepipeline_webhook
cognitoaws_cognito_identity_poolaws_cognito_user_pool
customer_gatewayaws_customer_gateway
configaws_config_config_ruleaws_config_configuration_recorderaws_config_delivery_channel
datapipelineaws_datapipeline_pipeline
devicefarmaws_devicefarm_project
dynamodbaws_dynamodb_table
ec2_instanceaws_instance
eipaws_eip
elasticacheaws_elasticache_clusteraws_elasticache_parameter_groupaws_elasticache_subnet_groupaws_elasticache_replication_group
ebsaws_ebs_volumeaws_volume_attachment
elastic_beanstalkaws_elastic_beanstalk_applicationaws_elastic_beanstalk_environment
ecsaws_ecs_clusteraws_ecs_serviceaws_ecs_task_definition
ecraws_ecr_lifecycle_policyaws_ecr_repositoryaws_ecr_repository_policy
efsaws_efs_access_pointaws_efs_file_systemaws_efs_file_system_policyaws_efs_mount_target
eksaws_eks_cluster
elbaws_elb
emraws_emr_clusteraws_emr_security_configuration
eniaws_network_interface
esaws_elasticsearch_domain
firehoseaws_kinesis_firehose_delivery_stream
glueglue_crawleraws_glue_catalog_databaseaws_glue_catalog_table
iamaws_iam_groupaws_iam_group_policyaws_iam_group_policy_attachmentaws_iam_instance_profileaws_iam_policyaws_iam_roleaws_iam_role_policyaws_iam_role_policy_attachmentaws_iam_useraws_iam_user_group_membershipaws_iam_user_policyaws_iam_user_policy_attachment
igwaws_internet_gateway
iotaws_iot_thingaws_iot_thing_typeaws_iot_topic_ruleaws_iot_role_alias
kinesisaws_kinesis_stream
kmsaws_kms_keyaws_kms_alias
lambdaaws_lambda_event_source_mappingaws_lambda_functionaws_lambda_function_event_invoke_configaws_lambda_layer_version
logsaws_cloudwatch_log_group
media_packageaws_media_package_channel
media_storeaws_media_store_container
mskaws_msk_cluster
nataws_nat_gateway
naclaws_network_acl
organizationaws_organizations_accountaws_organizations_organizationaws_organizations_organizational_unitaws_organizations_policyaws_organizations_policy_attachment
qldbaws_qldb_ledger
rdsaws_db_instanceaws_db_parameter_groupaws_db_subnet_groupaws_db_option_groupaws_db_event_subscription
resourcegroupsaws_resourcegroups_group
route53aws_route53_zoneaws_route53_record
route_tableaws_route_tableaws_main_route_table_associationaws_route_table_association
s3aws_s3_bucketaws_s3_bucket_policy
secretsmanageraws_secretsmanager_secret
securityhubaws_securityhub_accountaws_securityhub_memberaws_securityhub_standards_subscription
servicecatalogaws_servicecatalog_portfolio
sesaws_ses_configuration_setaws_ses_domain_identityaws_ses_email_identityaws_ses_receipt_ruleaws_ses_receipt_rule_setaws_ses_template
sfnaws_sfn_activityaws_sfn_state_machine
sgaws_security_groupaws_security_group_rule(if a rule cannot be inlined)
snsaws_sns_topicaws_sns_topic_subscription
sqsaws_sqs_queue
subnetaws_subnet
swfaws_swf_domain
transit_gatewayaws_ec2_transit_gateway_route_tableaws_ec2_transit_gateway_vpc_attachment
wafaws_waf_byte_match_setaws_waf_geo_match_setaws_waf_ipsetaws_waf_rate_based_ruleaws_waf_regex_match_setaws_waf_regex_pattern_setaws_waf_ruleaws_waf_rule_groupaws_waf_size_constraint_setaws_waf_sql_injection_match_setaws_waf_web_aclaws_waf_xss_match_set
waf_regionalaws_wafregional_byte_match_setaws_wafregional_geo_match_setaws_wafregional_ipsetaws_wafregional_rate_based_ruleaws_wafregional_regex_match_setaws_wafregional_regex_pattern_setaws_wafregional_ruleaws_wafregional_rule_groupaws_wafregional_size_constraint_setaws_wafregional_sql_injection_match_setaws_wafregional_web_aclaws_wafregional_xss_match_set
vpcaws_vpc
vpc_peeringaws_vpc_peering_connection
vpn_connectionaws_vpn_connection
vpn_gatewayaws_vpn_gateway
workspacesaws_workspaces_directoryaws_workspaces_ip_groupaws_workspaces_workspace
xrayaws_xray_sampling_rule
โ Global services
AWS services that are global will be imported without specified region even if several regions will be passed. It is to ensure only one representation of an AWS resource is imported.
List of global AWS services:
budgetscloudfrontiamorganizationroute53waf
โ Attribute filters
Attribute filters allow filtering across different resource types by its attributes.
terraformer import aws --resources=ec2_instance,ebs --filter="Name=tags.costCenter;Value=20000:'20001:1'" --regions=eu-west-1
Will only import AWS EC2 instances along with EBS volumes annotated with tag costCenter with values 20000 or 20001:1. Attribute filters are by default applicable to all resource types although it's possible to specify to what resource type a given filter should be applicable to by providing Type=<type> parameter. For example:
terraformer import aws --resources=ec2_instance,ebs --filter=Type=ec2_instance;Name=tags.costCenter;Value=20000:'20001:1' --regions=eu-west-1
Will work as same as example above with a change the filter w
Tag summary
Content type
Image
Digest
Size
993 MB
Last updated
over 4 years ago
docker pull bridgecrew/terraformer