Add to Docker Desktop
Version 4.43 or later needs to be installed to add the server automatically
Tools
| Name | Description |
|---|---|
add_inline_policy | Add a new inline policy to an IAM role. This tool creates a new inline policy with the specified permissions and adds it to an IAM role. Inline policies are embedded within the role and cannot be attached to multiple roles. Commonly used for granting data processing services access to AWS resources, enabling Glue jobs to access data sources, and configuring permissions for CloudWatch logging and S3 access. ## Requirements - The server must be run with the `--allow-write` flag - The role must exist in your AWS account - The policy name must be unique within the role - You cannot modify existing policies with this tool ## Permission Format The permissions parameter can be either a single policy statement or a list of statements. ### Single Statement Example ```json { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::example-bucket/*" } ``` ## Common Data Processing Permission Examples ### Glue Job Permissions ```json { "Effect": "Allow", "Action": [ "glue:*", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "iam:PassRole" ], "Resource": "*" } ``` ### EMR Cluster Permissions ```json { "Effect": "Allow", "Action": [ "elasticmapreduce:*", "ec2:DescribeInstances", "ec2:DescribeSecurityGroups", "s3:ListBucket", "s3:GetObject", "s3:PutObject" ], "Resource": "*" } ``` ### Athena Query Permissions ```json { "Effect": "Allow", "Action": [ "athena:*", "glue:GetDatabase", "glue:GetTable", "glue:GetPartition", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ], "Resource": "*" } ``` ## Usage Tips - Follow the principle of least privilege by granting only necessary permissions - Use specific resources rather than "*" whenever possible - Consider using conditions to further restrict permissions - Group related permissions into logical policies with descriptive names |
analyze_s3_usage_for_data_processing | Analyze S3 bucket usage patterns for data processing services (Glue, EMR, Athena). This tool helps identify which buckets are actively used by data processing services and which ones might be idle or underutilized. |
create_data_processing_role | Create a new IAM role for data processing services. This tool creates a new IAM role with the appropriate trust relationship for the specified data processing service (Glue, EMR, or Athena). It can also attach managed policies and add an inline policy to the role. ## Requirements - The server must be run with the `--allow-write` flag - The role name must be unique within your AWS account - Valid AWS credentials with permissions to create IAM roles ## Service Types - **glue**: Creates a role that can be assumed by the Glue service - **emr**: Creates a role that can be assumed by the EMR service - **athena**: Creates a role that can be assumed by the Athena service ## Common Managed Policies. add these policies - Glue: 'arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole' - EMR: 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole' - Athena: 'arn:aws:iam::aws:policy/service-role/AmazonAthenaFullAccess' ## Usage Tips - Always provide a descriptive name and description for the role - Attach only the necessary managed policies to follow least privilege - Use inline policies for custom permissions specific to your use case - Consider adding S3 access permissions for data sources and targets |
get_policies_for_role | Get all policies attached to an IAM role. This tool retrieves all policies associated with an IAM role, providing a comprehensive view of the role's permissions and trust relationships. It helps you understand the current permissions, identify missing or excessive permissions, troubleshoot data processing issues, and verify trust relationships for service roles. ## Requirements - The role must exist in your AWS account - Valid AWS credentials with permissions to read IAM role information ## Response Information The response includes role ARN, assume role policy document (trust relationships), role description, managed policies with their documents, and inline policies with their documents. ## Usage Tips - Use this tool before adding new permissions to understand existing access - Check the assume role policy to verify which services or roles can assume this role - Look for overly permissive policies that might pose security risks - Use with add_inline_policy to implement least-privilege permissions - For Glue jobs, ensure the role has access to required data sources and targets - For EMR clusters, verify EC2 instance profile permissions - For Athena queries, check S3 bucket access permissions |
get_roles_for_service | Get all IAM roles that can be assumed by a specific AWS service. This tool retrieves all IAM roles in your AWS account that have a trust relationship with the specified service. It helps you identify which roles can be used for services like Glue jobs, EMR clusters, or Athena queries, making it easier to select the appropriate role when creating these resources. ## Service Types Common service types include: - **glue**: AWS Glue service (glue.amazonaws.com) - **emr**: Amazon EMR service (elasticmapreduce.amazonaws.com) - **athena**: Amazon Athena service (athena.amazonaws.com) - You can also specify other AWS service principals ## Response Information The response includes a list of roles that can be assumed by the specified service, with details such as role name, ARN, description, creation date, and the full assume role policy document. ## Usage Tips - Use this tool to find existing roles before creating new ones - Verify that roles have the necessary permissions for your use case - For Glue jobs, look for roles with AWSGlueServiceRole or similar policies - For EMR clusters, look for roles with AmazonElasticMapReduceRole or similar policies - For Athena queries, look for roles with AmazonAthenaFullAccess or similar policies |
list_s3_buckets | List S3 buckets that have 'glue' in their name and are in the specified region. This tool helps identify S3 buckets commonly used for data processing workflows, particularly those related to AWS Glue operations. It provides usage statistics and idle time information to help with resource management. ## Requirements - Valid AWS credentials with permissions to list S3 buckets - S3:ListAllMyBuckets permission ## Response Information The response includes bucket name, creation date, region, object count, last modified date, and idle time analysis. ## Usage Tips - Use this tool to find existing data processing buckets before creating new ones - Monitor idle buckets that haven't been accessed for 90+ days - Verify bucket regions match your data processing service regions - Check object counts to understand bucket usage patterns |
manage_aws_athena_data_catalogs | Manage AWS Athena data catalogs with both read and write operations. This tool provides operations for managing Athena data catalogs, including creating, retrieving, listing, updating, and deleting data catalogs. Data catalogs are used to organize and access data sources in Athena, enabling you to query data across various sources like AWS Glue Data Catalog, Apache Hive metastores, or federated sources. ## Requirements - The server must be run with the `--allow-write` flag for create-data-catalog, delete-data-catalog, and update-data-catalog operations - Appropriate AWS permissions for Athena data catalog operations ## Operations - **create-data-catalog**: Create a new data catalog - **delete-data-catalog**: Delete an existing data catalog - **get-data-catalog**: Get information about a single data catalog - **list-data-catalogs**: List all data catalogs - **update-data-catalog**: Update an existing data catalog ## Usage Tips - Use list-data-catalogs to find available data catalogs - Data catalogs can be of type LAMBDA, GLUE, HIVE, or FEDERATED - Parameters are specific to the type of data catalog ## Example ``` # List all data catalogs {'operation': 'list-data-catalogs', 'max_results': 10} # Create a Glue data catalog { 'operation': 'create-data-catalog', 'name': 'my-glue-catalog', 'type': 'GLUE', 'description': 'My Glue Data Catalog', 'parameters': {'catalog-id': '123456789012'}, } ``` |
manage_aws_athena_databases_and_tables | Manage AWS Athena databases and tables with read operations. This tool provides operations for retrieving information about databases and tables in Athena data catalogs. These are read-only operations that do not modify any resources. ## Requirements - Appropriate AWS permissions for Athena database and table operations ## Operations - **get-database**: Get information about a single database - **get-table-metadata**: Get metadata for a specific table - **list-databases**: List all databases in a data catalog - **list-table-metadata**: List metadata for all tables in a database ## Usage Tips - Use list-databases to find available databases in a data catalog - Use list-table-metadata to find available tables in a database - The expression parameter for list-table-metadata supports filtering tables by name pattern ## Example ``` # List all databases in a catalog {'operation': 'list-databases', 'catalog_name': 'AwsDataCatalog', 'max_results': 10} # Get metadata for a specific table { 'operation': 'get-table-metadata', 'catalog_name': 'AwsDataCatalog', 'database_name': 'my_database', 'table_name': 'my_table', } ``` |
manage_aws_athena_named_queries | Manage saved SQL queries in AWS Athena. This tool provides operations for creating, retrieving, updating, and deleting named queries in AWS Athena. Named queries are saved SQL statements that can be easily reused, shared with team members, and executed without having to rewrite complex queries. ## Requirements - The server must be run with the `--allow-write` flag for create-named-query, delete-named-query, and update-named-query operations - Appropriate AWS permissions for Athena named query operations ## Operations - **batch-get-named-query**: Get details for up to 50 named queries by their IDs - **create-named-query**: Save a new SQL query with a name and description - **delete-named-query**: Remove a saved query - **get-named-query**: Retrieve a single named query by ID - **list-named-queries**: List available named query IDs - **update-named-query**: Modify an existing named query ## Example ```python # Create a named query create_response = await manage_aws_athena_named_queries( operation='create-named-query', name='Daily Active Users', description='Query to calculate daily active users', database='analytics', query_string='SELECT date, COUNT(DISTINCT user_id) AS active_users FROM user_events GROUP BY date ORDER BY date DESC', work_group='primary', ) # Later, retrieve the named query query = await manage_aws_athena_named_queries( operation='get-named-query', named_query_id=create_response.named_query_id ) ``` |
manage_aws_athena_query_executions | Execute and manage AWS Athena SQL queries. This tool provides comprehensive operations for AWS Athena query management, including starting new queries, monitoring execution status, retrieving results, and analyzing performance statistics. ## Requirements - The server must be run with the `--allow-write` flag if start-query-execution contains any write operation for example DDL commands, Insert, Update, Delete Commands or any flag updates - Appropriate AWS permissions for Athena query operations ## Operations - **batch-get-query-execution**: Get details for up to 50 query executions by their IDs - **get-query-execution**: Get complete information about a single query execution - **get-query-results**: Retrieve the results of a completed query - **get-query-runtime-statistics**: Get performance statistics for a query execution - **list-query-executions**: List available query execution IDs (up to 50) - **start-query-execution**: Execute a new SQL query - **stop-query-execution**: Cancel a running query ## Example ```python # Start a new query response = await manage_aws_athena_queries( operation='start-query-execution', query_string='SELECT * FROM my_database.my_table LIMIT 10', query_execution_context={'Database': 'my_database', 'Catalog': 'my_catalog'}, work_group='primary', ) # Get the query results results = await manage_aws_athena_queries( operation='get-query-results', query_execution_id=response.query_execution_id ) ``` |
manage_aws_athena_workgroups | Manage AWS Athena workgroups with both read and write operations. This tool provides operations for managing Athena workgroups, including creating, retrieving, listing, updating, and deleting workgroups. Workgroups allow you to isolate queries for different user groups and control query execution settings. ## Requirements - The server must be run with the `--allow-write` flag for create-work-group, delete-work-group, and update-work-group operations - Appropriate AWS permissions for Athena workgroup operations ## Operations - **create-work-group**: Create a new workgroup - **delete-work-group**: Delete an existing workgroup - **get-work-group**: Get information about a single workgroup - **list-work-groups**: List all workgroups - **update-work-group**: Update an existing workgroup ## Usage Tips - Use workgroups to isolate different user groups and control costs - Configure workgroup settings to enforce query limits and output locations - Use tags to organize and track workgroups |
manage_aws_emr_clusters | Manage AWS EMR EC2 clusters with comprehensive control over cluster lifecycle. This tool provides operations for managing Amazon EMR clusters running on EC2 instances, including creating, configuring, monitoring, modifying, and terminating clusters. It also supports security configuration management for EMR clusters. ## Requirements - The server must be run with the `--allow-write` flag for create-cluster, modify-cluster, modify-cluster-attributes, terminate-clusters, create-security-configuration, and delete-security-configuration operations - Appropriate AWS permissions for EMR cluster operations ## Operations - **create-cluster**: Create a new EMR cluster with specified configurations - **describe-cluster**: Get detailed information about a specific EMR cluster - **modify-cluster**: Modify the step concurrency level of a running cluster - **modify-cluster-attributes**: Modify auto-termination and termination protection settings - **terminate-clusters**: Terminate one or more EMR clusters - **list-clusters**: List all EMR clusters with optional filtering - **create-security-configuration**: Create a new EMR security configuration - **delete-security-configuration**: Delete an existing EMR security configuration - **describe-security-configuration**: Get details about a specific security configuration - **list-security-configurations**: List all available security configurations ## Example ``` # Create a basic EMR cluster with Spark { 'operation': 'create-cluster', 'name': 'SparkCluster', 'release_label': 'emr-7.9.0', 'applications': [{'Name': 'Spark'}], 'instances': { 'InstanceGroups': [ { 'Name': 'Master', 'InstanceRole': 'MASTER', 'InstanceType': 'm5.xlarge', 'InstanceCount': 1, }, { 'Name': 'Core', 'InstanceRole': 'CORE', 'InstanceType': 'm5.xlarge', 'InstanceCount': 2, }, ], 'Ec2KeyName': 'my-key-pair', 'KeepJobFlowAliveWhenNoSteps': true, }, } ``` ## Usage Tips - Use list-clusters to find cluster IDs before performing operations on specific clusters - Check cluster state before performing operations that require specific states - For large result sets, use pagination with marker parameter - When creating clusters, consider using security configurations for encryption and authentication |
manage_aws_emr_ec2_instances | Manage AWS EMR EC2 instances with both read and write operations. This tool provides comprehensive operations for managing Amazon EMR EC2 instances, including adding and modifying instance fleets and groups, as well as listing instance details. It enables scaling cluster capacity, configuring instance specifications, and monitoring instance status. ## Requirements - The server must be run with the `--allow-write` flag for add-instance-fleet, add-instance-groups, modify-instance-fleet, and modify-instance-groups operations - Appropriate AWS permissions for EMR instance operations ## Operations - **add-instance-fleet**: Add an instance fleet to an existing EMR cluster - Required: cluster_id, instance_fleet (with InstanceFleetType) - Returns: cluster_id, instance_fleet_id, cluster_arn - **add-instance-groups**: Add instance groups to an existing EMR cluster - Required: cluster_id, instance_groups (each with InstanceRole, InstanceType, InstanceCount) - Returns: cluster_id (as job_flow_id), instance_group_ids, cluster_arn - **modify-instance-fleet**: Modify an instance fleet in an EMR cluster - Required: cluster_id, instance_fleet_id, instance_fleet_config - Returns: confirmation of modification - **modify-instance-groups**: Modify instance groups in an EMR cluster - Required: instance_group_configs (each with InstanceGroupId) - Optional: cluster_id - Returns: confirmation of modification - **list-instance-fleets**: List all instance fleets in an EMR cluster - Required: cluster_id - Optional: marker - Returns: instance_fleets, marker for pagination - **list-instances**: List all instances in an EMR cluster - Required: cluster_id - Optional: instance_group_id, instance_group_types, instance_fleet_id, instance_fleet_type, instance_states, marker - Returns: instances, marker for pagination - **list-supported-instance-types**: List all supported instance types for EMR - Required: release_label - Optional: marker - Returns: instance_types, marker for pagination ## Example ```python # Add a task instance fleet with mixed instance types response = await manage_aws_emr_ec2_instances( operation='add-instance-fleet', cluster_id='j-123ABC456DEF', instance_fleet={ 'InstanceFleetType': 'TASK', 'Name': 'TaskFleet', 'TargetOnDemandCapacity': 2, 'TargetSpotCapacity': 3, 'InstanceTypeConfigs': [ { 'InstanceType': 'm5.xlarge', 'WeightedCapacity': 1, 'BidPriceAsPercentageOfOnDemandPrice': 80, }, { 'InstanceType': 'm5.2xlarge', 'WeightedCapacity': 2, 'BidPriceAsPercentageOfOnDemandPrice': 75, }, ], }, ) ``` |
manage_aws_emr_ec2_steps | Manage AWS EMR EC2 steps for processing data on EMR clusters. This tool provides comprehensive operations for managing EMR steps, which are units of work submitted to an EMR cluster for execution. Steps typically consist of Hadoop or Spark jobs that process and analyze data. ## Requirements - The server must be run with the `--allow-write` flag for add-steps and cancel-steps operations - Appropriate AWS permissions for EMR step operations ## Operations - **add-steps**: Add new steps to a running EMR cluster (max 256 steps per job flow) - **cancel-steps**: Cancel pending or running steps on an EMR cluster (EMR 4.8.0+ except 5.0.0) - **describe-step**: Get detailed information about a specific step's configuration and status - **list-steps**: List and filter steps for an EMR cluster with pagination support ## Usage Tips - Each step consists of a JAR file, its main class, and arguments - Steps are executed in the order listed and must exit with zero code to be considered complete - For cancel-steps, you can specify SEND_INTERRUPT (default) or TERMINATE_PROCESS as cancellation option - When listing steps, filter by step states: PENDING, CANCEL_PENDING, RUNNING, COMPLETED, CANCELLED, FAILED, INTERRUPTED - For large result sets, use pagination with marker parameter ## Example ``` # Add a Spark step to process data { 'operation': 'add-steps', 'cluster_id': 'j-2AXXXXXXGAPLF', 'steps': [ { 'Name': 'Spark Data Processing', 'ActionOnFailure': 'CONTINUE', 'HadoopJarStep': { 'Jar': 'command-runner.jar', 'Args': [ 'spark-submit', '--class', 'com.example.SparkProcessor', 's3://mybucket/myapp.jar', 'arg1', 'arg2', ], }, } ], } ``` |
manage_aws_glue_catalog | Manage AWS Glue Data Catalog with both read and write operations. This tool provides operations for managing the Glue Data Catalog itself, including creating custom catalogs, importing from external sources, and managing catalog-level configurations. ## Requirements - The server must be run with the `--allow-write` flag for create-catalog, delete-catalog, and import operations - Appropriate AWS permissions for Glue Data Catalog operations - For import operations, access to the external data source is required ## Operations - **create-catalog**: Create a new data catalog - **delete-catalog**: Delete an existing data catalog - **get-catalog**: Retrieve detailed information about a specific catalog - **list-catalogs**: List all available catalogs - **import-catalog-to-glue**: Import metadata from external sources into Glue Data Catalog ## Usage Tips - The default catalog ID is your AWS account ID - Custom catalogs allow for better organization and access control - Import operations can take significant time depending on source size |
manage_aws_glue_classifiers | Manage AWS Glue classifiers to determine data formats and schemas. This tool provides operations for AWS Glue classifiers, which help determine the schema of your data. Classifiers analyze data samples to infer formats and structures, enabling accurate schema creation when crawlers process your data sources. ## Requirements - The server must be run with the `--allow-write` flag for create, delete, and update operations - Appropriate AWS permissions for Glue classifier operations ## Operations - **create-classifier**: Create a new custom classifier (CSV, JSON, XML, or GROK) - **delete-classifier**: Remove an existing classifier - **get-classifier**: Retrieve detailed information about a specific classifier - **get-classifiers**: List all available classifiers - **update-classifier**: Modify an existing classifier's configuration ## Example ```python # Create a CSV classifier { 'operation': 'create-classifier', 'classifier_definition': { 'CsvClassifier': { 'Name': 'my-csv-classifier', 'Delimiter': ',', 'QuoteSymbol': '"', 'ContainsHeader': 'PRESENT', 'Header': ['id', 'name', 'date', 'value'], 'AllowSingleColumn': false, } }, } ``` |
manage_aws_glue_connections | Manage AWS Glue Data Catalog connections with both read and write operations. Connections in AWS Glue store connection information for data stores, such as databases, data warehouses, and other data sources. They contain connection properties like JDBC URLs, usernames, and other metadata needed to connect to external data sources. ## Requirements - The server must be run with the `--allow-write` flag for create, update, and delete operations - Appropriate AWS permissions for Glue Data Catalog operations - Connection properties must be valid for the connection type ## Operations - **create-connection**: Create a new connection - **delete-connection**: Delete an existing connection - **get-connection**: Retrieve detailed information about a specific connection - **list-connections**: List all connections - **update-connection**: Update an existing connection's properties ## Usage Tips - Connection names must be unique within your catalog - Connection input should include ConnectionType and ConnectionProperties - Use get or list operations to check existing connections before creating |
manage_aws_glue_crawler_management | Manage AWS Glue crawler schedules and monitor performance metrics. This tool provides operations for controlling crawler schedules and retrieving performance metrics. Use it to automate crawler runs on a schedule and monitor crawler efficiency and status. ## Requirements - The server must be run with the `--allow-write` flag for schedule management operations - Appropriate AWS permissions for Glue crawler operations ## Operations - **get-crawler-metrics**: Retrieve performance statistics about crawlers - **start-crawler-schedule**: Activate a crawler's schedule - **stop-crawler-schedule**: Deactivate a crawler's schedule - **update-crawler-schedule**: Modify a crawler's schedule with a new cron expression ## Example ```python # Update a crawler's schedule to run daily at 2:30 AM UTC { 'operation': 'update-crawler-schedule', 'crawler_name': 'my-s3-data-crawler', 'schedule': 'cron(30 2 * * ? *)', } # Get metrics for specific crawlers { 'operation': 'get-crawler-metrics', 'crawler_name_list': ['my-s3-data-crawler', 'my-jdbc-crawler'], } ``` |
manage_aws_glue_crawlers | Manage AWS Glue crawlers to discover and catalog data sources. This tool provides comprehensive operations for AWS Glue crawlers, which automatically discover and catalog data from various sources like S3, JDBC databases, DynamoDB, and more. Crawlers examine your data sources, determine schemas, and register metadata in the AWS Glue Data Catalog. ## Requirements - The server must be run with the `--allow-write` flag for create, delete, start, stop, and update operations - Appropriate AWS permissions for Glue crawler operations ## Operations - **create-crawler**: Create a new crawler with specified targets, role, and configuration - **delete-crawler**: Remove an existing crawler from AWS Glue - **get-crawler**: Retrieve detailed information about a specific crawler - **get-crawlers**: List all crawlers with pagination - **batch-get-crawlers**: Retrieve multiple specific crawlers in a single call - **list-crawlers**: List all crawlers with tag-based filtering - **start-crawler**: Initiate a crawler run immediately - **stop-crawler**: Halt a currently running crawler - **update-crawler**: Modify an existing crawler's configuration ## Example ```python # Create a new S3 crawler { 'operation': 'create-crawler', 'crawler_name': 'my-s3-data-crawler', 'crawler_definition': { 'Role': 'arn:aws:iam::123456789012:role/GlueServiceRole', 'Targets': {'S3Targets': [{'Path': 's3://my-bucket/data/'}]}, 'DatabaseName': 'my_catalog_db', 'Description': 'Crawler for S3 data files', 'Schedule': 'cron(0 0 * * ? *)', 'TablePrefix': 'raw_', }, } ``` |
manage_aws_glue_databases | Manage AWS Glue Data Catalog databases with both read and write operations. This tool provides operations for managing Glue Data Catalog databases, including creating, updating, retrieving, listing, and deleting databases. It serves as the primary mechanism for database management within the AWS Glue Data Catalog. ## Requirements - The server must be run with the `--allow-write` flag for create-database, update-database, and delete-database operations - Appropriate AWS permissions for Glue Data Catalog operations ## Operations - **create-database**: Create a new database in the Glue Data Catalog - **delete-database**: Delete an existing database from the Glue Data Catalog - **get-database**: Retrieve detailed information about a specific database - **list-databases**: List all databases in the Glue Data Catalog - **update-database**: Update an existing database's properties ## Usage Tips - Use the get-database or list-databases operations first to check existing databases - Database names must be unique within your AWS account and region - Deleting a database will also delete all tables within it |
manage_aws_glue_encryption | Manage AWS Glue Data Catalog Encryption Settings for data protection. This tool allows you to retrieve and update AWS Glue Data Catalog Encryption Settings, which control how metadata and connection passwords are encrypted in the Data Catalog. ## Requirements - The server must be run with the `--allow-write` flag for put-catalog-encryption-settings operation - Appropriate AWS permissions for Glue Data Catalog Encryption operations ## Operations - **get-catalog-encryption-settings**: Retrieve the current encryption settings for the Data Catalog - **put-catalog-encryption-settings**: Update the encryption settings for the Data Catalog ## Example ```json { "operation": "put-catalog-encryption-settings", "encryption_at_rest": { "CatalogEncryptionMode": "SSE-KMS", "SseAwsKmsKeyId": "arn:aws:kms:region:account-id:key/key-id" }, "connection_password_encryption": { "ReturnConnectionPasswordEncrypted": true, "AwsKmsKeyId": "arn:aws:kms:region:account-id:key/key-id" } } ``` |
manage_aws_glue_jobs | Manage AWS Glue ETL jobs and job runs with both read and write operations. This tool provides comprehensive operations for managing AWS Glue ETL jobs and job runs, including creating, updating, retrieving, listing, starting, stopping, and monitoring jobs. ## Requirements - The server must be run with the `--allow-write` flag for create-job, delete-job, update-job, start-job-run, stop-job-run, and batch-stop-job-run operations - Appropriate AWS permissions for Glue ETL job operations ## Job Operations - **create-job**: Create a new ETL job in AWS Glue - **delete-job**: Delete an existing ETL job from AWS Glue - **get-job**: Retrieve detailed information about a specific job - **get-jobs**: List all jobs in your AWS Glue account - **update-job**: Update an existing job's properties - **start-job-run**: Start a job run using a job name ## Job Run Operations - **stop-job-run**: Stop a job run using a job name and run ID - **get-job-run**: Retrieve detailed information about a specific job run - **get-job-runs**: List all job runs for a specific job - **batch-stop-job-run**: Stop one or more running jobs ## Usage Tips - Job names must be unique within your AWS account and region - Create a script required by the customer and push the script to a customer S3 Location. Ask for S3 Location if not provided. - Verify if the IAM role used has glue trusted entities in the role if not update the role or create a new one - Job definitions should include command, role, and other required parameters - As rule of thumb use Glue Version 5.0 or latest to create jobs ## Examples ``` # Create a new Spark ETL job { 'operation': 'create-job', 'job_name': 'my-etl-job', 'job_definition': { 'Role': 'arn:aws:iam::123456789012:role/GlueETLRole', 'Command': { 'Name': 'glueetl', 'ScriptLocation': 's3://my-bucket/scripts/etl-script.py', }, 'GlueVersion': '5.0', 'MaxRetries': 2, 'Timeout': 120, 'WorkerType': 'G.1X', 'NumberOfWorkers': 5, }, } # Start a job run { 'operation': 'start-job-run', 'job_name': 'my-etl-job', 'worker_type': 'G.1X', 'number_of_workers': 5, } # Get details of a specific job run { 'operation': 'get-job-run', 'job_name': 'my-etl-job', 'job_run_id': 'jr_1234567890abcdef0', } ``` |
manage_aws_glue_partitions | Manage AWS Glue Data Catalog partitions with both read and write operations. Partitions in AWS Glue represent a way to organize table data based on the values of one or more columns. They enable efficient querying and processing of large datasets by allowing queries to target specific subsets of data. ## Requirements - The server must be run with the `--allow-write` flag for create-partition, update-partition, and delete-partition operations - Database and table must exist before creating partitions - Partition values must match the partition schema defined in the table ## Operations - **create-partition**: Create a new partition in the specified table - **delete-partition**: Delete an existing partition from the table - **get-partition**: Retrieve detailed information about a specific partition - **list-partitions**: List all partitions in the specified table - **update-partition**: Update an existing partition's properties ## Usage Tips - Partition values must be provided in the same order as partition columns in the table - Use get-partition or list-partitions operations to check existing partitions before creating - Partition input should include storage descriptor and location information |
manage_aws_glue_resource_policies | Manage AWS Glue Resource Policies for access control. This tool allows you to retrieve, create, update, and delete AWS Glue Resource Policies, which control access to Glue resources through IAM policy documents. ## Requirements - The server must be run with the `--allow-write` flag for put-resource-policy and delete-resource-policy operations - Appropriate AWS permissions for Glue Resource Policy operations ## Operations - **get-resource-policy**: Retrieve the current resource policy - **put-resource-policy**: Create or update the resource policy - **delete-resource-policy**: Delete the resource policy ## Example ```json { "operation": "put-resource-policy", "policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::123456789… "policy_exists_condition": "NOT_EXIST", "enable_hybrid": true } ``` |
manage_aws_glue_security_configurations | Manage AWS Glue Security Configurations for data encryption. This tool allows you to create, retrieve, and delete AWS Glue Security Configurations, which define encryption settings for Glue jobs, crawlers, and development endpoints. ## Requirements - The server must be run with the `--allow-write` flag for create-security-configuration and delete-security-configuration operations - Appropriate AWS permissions for Glue Security Configuration operations ## Operations - **create-security-configuration**: Create a new security configuration with encryption settings - **delete-security-configuration**: Delete an existing security configuration - **get-security-configuration**: Retrieve detailed information about a specific security configuration ## Example ```json { "operation": "create-security-configuration", "config_name": "my-encryption-config", "encryption_configuration": { "S3Encryption": [ { "S3EncryptionMode": "SSE-KMS", "KmsKeyArn": "arn:aws:kms:region:account-id:key/key-id" } ], "CloudWatchEncryption": { "CloudWatchEncryptionMode": "DISABLED" }, "JobBookmarksEncryption": { "JobBookmarksEncryptionMode": "CSE-KMS", "KmsKeyArn": "arn:aws:kms:region:account-id:key/key-id" } } } ``` |
manage_aws_glue_sessions | Manage AWS Glue Interactive Sessions for running Spark and Ray workloads. This tool provides operations for creating and managing Glue Interactive Sessions, which enable interactive development and execution of Spark ETL scripts and Ray applications. Interactive sessions provide a responsive environment for data exploration, debugging, and iterative development. ## Requirements - The server must be run with the `--allow-write` flag for create-session, delete-session, and stop-session operations - Appropriate AWS permissions for Glue Interactive Session operations ## Operations - **create-session**: Create a new interactive session with specified configuration - **delete-session**: Delete an existing interactive session - **get-session**: Retrieve detailed information about a specific session - **list-sessions**: List all interactive sessions with optional filtering - **stop-session**: Stop a running interactive session ## Example ```python # Create a new Spark ETL session { 'operation': 'create-session', 'session_id': 'my-spark-session', 'role': 'arn:aws:iam::123456789012:role/GlueInteractiveSessionRole', 'command': {'Name': 'glueetl', 'PythonVersion': '3'}, 'glue_version': '3.0', } ``` |
manage_aws_glue_statements | Manage AWS Glue Interactive Session Statements for executing code and retrieving results. This tool provides operations for executing code, canceling running statements, and retrieving results within Glue Interactive Sessions. It enables interactive data processing, exploration, and analysis using Spark or Ray in AWS Glue. ## Requirements - The server must be run with the `--allow-write` flag for run-statement and cancel-statement operations - Appropriate AWS permissions for Glue Interactive Session Statement operations - A valid session ID is required for all operations ## Operations - **run-statement**: Execute code in an interactive session and get a statement ID - **cancel-statement**: Cancel a running statement by ID - **get-statement**: Retrieve detailed information and results of a specific statement - **list-statements**: List all statements in a session with their status ## Example ```python # Run a PySpark statement in a session { 'operation': 'run-statement', 'session_id': 'my-spark-session', 'code': "df = spark.read.csv('s3://my-bucket/data.csv', header=True)\ndf.show(5)", } ``` |
manage_aws_glue_tables | Manage AWS Glue Data Catalog tables with both read and write operations. This tool provides comprehensive operations for managing Glue Data Catalog tables, including creating, updating, retrieving, listing, searching, and deleting tables. Tables define the schema and metadata for data stored in various formats and locations. ## Requirements - The server must be run with the `--allow-write` flag for create-table, update-table, and delete-table operations - Database must exist before creating tables within it - Appropriate AWS permissions for Glue Data Catalog operations ## Operations - **create-table**: Create a new table in the specified database - **delete-table**: Delete an existing table from the database - **get-table**: Retrieve detailed information about a specific table - **list-tables**: List all tables in the specified database - **update-table**: Update an existing table's properties - **search-tables**: Search for tables using text matching ## Usage Tips - Table names must be unique within a database - Use get-table or list-tables operations to check existing tables before creating - Table input should include storage descriptor, columns, and partitioning information |
manage_aws_glue_triggers | Manage AWS Glue triggers to automate workflow and job execution. This tool allows you to create, delete, retrieve, list, start, and stop AWS Glue triggers. Triggers define the conditions that automatically start jobs or workflows, enabling scheduled or event-based execution of your ETL processes. ## Requirements - The server must be run with the `--allow-write` flag for create-trigger, delete-trigger, start-trigger, and stop-trigger operations - Appropriate AWS permissions for Glue trigger operations ## Operations - **create-trigger**: Create a new trigger with specified type (SCHEDULED, CONDITIONAL, ON_DEMAND, EVENT) and actions - **delete-trigger**: Delete an existing trigger by name - **get-trigger**: Retrieve detailed information about a specific trigger - **get-triggers**: List all triggers with pagination support - **start-trigger**: Activate a trigger to begin monitoring for its firing conditions - **stop-trigger**: Deactivate a trigger to pause its monitoring ## Trigger Types - **SCHEDULED**: Time-based triggers that run on a cron schedule - **CONDITIONAL**: Event-based triggers that run when specified conditions are met - **ON_DEMAND**: Manually activated triggers - **EVENT**: EventBridge event-based triggers ## Example ```python # Create a scheduled trigger manage_aws_glue_triggers( operation='create-trigger', trigger_name='daily-etl-trigger', trigger_definition={ 'Type': 'SCHEDULED', 'Schedule': 'cron(0 12 * * ? *)', # Run daily at 12:00 UTC 'Actions': [{'JobName': 'process-daily-data'}], 'Description': 'Trigger for daily ETL job', 'StartOnCreation': True, }, ) # Create a conditional trigger manage_aws_glue_triggers( operation='create-trigger', trigger_name='data-arrival-trigger', trigger_definition={ 'Type': 'CONDITIONAL', 'Actions': [{'JobName': 'process-new-data'}], 'Predicate': { 'Conditions': [ { 'LogicalOperator': 'EQUALS', 'JobName': 'crawl-new-data', 'State': 'SUCCEEDED', } ] }, 'Description': 'Trigger that runs when data crawling completes', }, ) ``` |
manage_aws_glue_usage_profiles | Manage AWS Glue Usage Profiles for resource allocation and cost management. This tool allows you to create, retrieve, update, and delete AWS Glue Usage Profiles, which define resource allocation and cost management settings for Glue jobs and interactive sessions. ## Requirements - The server must be run with the `--allow-write` flag for create-profile, delete-profile, and update-profile operations - Appropriate AWS permissions for Glue Usage Profile operations ## Operations - **create-profile**: Create a new usage profile with specified resource allocations - **delete-profile**: Delete an existing usage profile - **get-profile**: Retrieve detailed information about a specific usage profile - **update-profile**: Update an existing usage profile's configuration ## Example ```json { "operation": "create-profile", "profile_name": "my-standard-profile", "description": "Standard resource allocation for ETL jobs", "configuration": { "JobConfiguration": { "numberOfWorkers": { "DefaultValue": "10", "MinValue": "1", "MaxValue": "10" }, "workerType": { "DefaultValue": "G.2X", "AllowedValues": [ "G.2X", "G.4X", "G.8X" ] }, } } ``` |
manage_aws_glue_workflows | Manage AWS Glue workflows to orchestrate complex ETL activities. This tool allows you to create, delete, retrieve, list, and start AWS Glue workflows. Workflows help you design and visualize complex ETL activities as a series of dependent jobs and crawlers, making it easier to manage and monitor your data processing pipelines. ## Requirements - The server must be run with the `--allow-write` flag for create-workflow, delete-workflow, and start-workflow-run operations - Appropriate AWS permissions for Glue workflow operations ## Operations - **create-workflow**: Create a new workflow with optional description, default run properties, tags, and max concurrent runs - **delete-workflow**: Delete an existing workflow by name - **get-workflow**: Retrieve detailed information about a specific workflow with optional graph inclusion - **list-workflows**: List all workflows with pagination support - **start-workflow-run**: Start a workflow run with optional run properties ## Example ```python # Create a new workflow manage_aws_glue_workflows( operation='create-workflow', workflow_name='my-etl-workflow', workflow_definition={ 'Description': 'ETL workflow for daily data processing', 'DefaultRunProperties': {'ENV': 'production'}, 'MaxConcurrentRuns': 1, }, ) # Start a workflow run manage_aws_glue_workflows( operation='start-workflow-run', workflow_name='my-etl-workflow', workflow_definition={'run_properties': {'EXECUTION_DATE': '2023-06-19'}}, ) ``` |
upload_to_s3 | Upload Python code content directly to an S3 bucket using putObject. This tool uploads Python code content directly to an S3 bucket, commonly used for storing Glue job scripts, EMR step scripts, or other data processing code. The uploaded file can be referenced by data processing services. ## Requirements - The server must be run with the `--allow-write` flag - Valid AWS credentials with permissions to write to the specified S3 bucket - The bucket must exist and be accessible ## Usage Tips - Use descriptive S3 keys that include version information or timestamps - Store scripts in organized folder structures (e.g., glue-jobs/, emr-steps/) - Consider using versioning on the S3 bucket for script history - The returned S3 URI can be used directly in Glue job configurations |