emptyset/cloudsift repository overview

⁠CloudSift - AWS Resource Scanner & Cost Optimizer

CloudSift is a powerful Go-based utility designed to optimize AWS infrastructure and reduce cloud costs by scanning resources across multiple AWS Organizations, Accounts, and Regions. It provides comprehensive insights into resource utilization and cost optimization opportunities.

⁠🚀 Key Features

⁠Comprehensive Resource Discovery

• Compute & Storage: Analyze EC2 instances, EBS volumes, snapshots, AMIs, and RDS instances
• Networking: Monitor Elastic IPs, Load Balancers, VPCs, and Security Groups
• Identity & Database: Track IAM users/roles, DynamoDB tables, and OpenSearch domains

⁠Advanced Cost Analysis

• Real-time cost estimation with AWS Pricing API integration
• Smart caching system for pricing data
• Detailed cost breakdowns and projections
• Resource lifetime calculations
• Multi-region pricing support

⁠Enterprise-Grade Performance

• Intelligent rate limiting with token bucket algorithm
• High-performance worker pool architecture
• Automatic rate adjustment and failure handling
• Optimized I/O and task distribution

⁠Rich Reporting

• Interactive HTML reports with modern UI
• Resource filtering and sorting capabilities
• Cost breakdown visualizations
• Detailed resource metadata
• Actionable optimization recommendations
• Multiple output formats (JSON, text logging)

⁠🐳 Docker Usage

CloudSift can be run as a Docker container with mounted volumes for reports and cache:

⁠Basic Commands

⁠Show help and available commands

docker run emptyset/cloudsift:latest help

⁠Initialize default configuration

docker run -v $(pwd):/app emptyset/cloudsift:latest init config

⁠List AWS accounts in your organization

docker run \
  -v $(pwd):/app \
  -v ~/.aws/:/app/.aws \
  emptyset/cloudsift:latest accounts list \
  --organization-role CloudSiftOrganizationRole

⁠Run a scan across all accounts and store results in s3

docker run \
  -v $(pwd):/app \
  -v ~/.aws/:/app/.aws \
  emptyset/cloudsift:latest scan \
  --organization-role CloudSiftOrganizationRole \
  --scanner-role CloudSiftScannerRole \
  --bucket your-bucket-name \
  --bucket-region us-west-2

⁠Run a scan on specific accounts

docker run \
  -v $(pwd):/app \
  -v ~/.aws/:/app/.aws \
  emptyset/cloudsift:latest scan \
  --organization-role CloudSiftOrganizationRole \
  --scanner-role CloudSiftScannerRole \
  --accounts 123456789012,987654321098

⁠Volume Mounts

• /app/reports: Directory where scan reports will be saved
• /app/cache: Directory for caching AWS pricing data
• /app/.aws: Mount your AWS credentials (read-only)

⁠Environment Variables

You can also use environment variables with the Docker container:

docker run \
  -v $(pwd):/app \
  -v ~/.aws/:/app/.aws \
  -e CLOUDSIFT_ORGANIZATION_ROLE="CloudSiftOrganizationRole" \
  -e CLOUDSIFT_SCANNER_ROLE="CloudSiftScannerRole" \
  -e CLOUDSIFT_BUCKET="your-bucket-name" \
  -e CLOUDSIFT_BUCKET_REGION="us-west-2" \
  emptyset/cloudsift:latest scan

⁠Tips

• Always mount the cache directory to avoid repeated AWS pricing API calls
• Use read-only mount for AWS credentials for better security
• Reports are generated in HTML and JSON formats in the mounted reports directory

⁠🔐 Security & Permissions

• Supports both single-account and multi-account scanning modes
• Automated CloudFormation setup for required IAM roles and S3 bucket
• Server-side encryption for scan results
• Comprehensive IAM policies for secure operation

⁠📊 Sample Output

View a sample CloudSift report here⁠ to see the tool in action.

⁠🔗 Links

• GitHub Repository⁠
• Documentation⁠
• Sample Report⁠

⁠License

This project is licensed under the Mozilla Public License 2.0 (MPL-2.0). For more details, refer to the license page⁠.

⁠Commercial License

For full proprietary use, premium features, or additional support, contact us at [email protected]⁠ for commercial licensing options.

⁠Contact

For questions and support, please email [email protected]⁠