dadosfera/base-kernel-py-agent repository overview

⁠Dadosfera Base Kernel Python Agent

This Dockerfile builds a custom Docker image for the Dadosfera platform, providing a Python-based environment tailored for data science and processing pipelines. It includes dependencies for Jupyter, AWS integration, Orchest workflow management, and Dadosfera-specific tools.

⁠Features

• Base Image: Built on top of jupyter/base-notebook:2022-03-09 (Ubuntu 20.04.1 LTS Focal)
• Multi-Environment Support: Includes both base conda environment and isolated Python 3.11 environment
• Enterprise Gateway Support: Includes kernel files and dependencies for Jupyter Enterprise Gateway v2.5.2
• Orchest Integration: Full support for Orchest workflow orchestration platform
• AI Agent Wrapper: Custom Dadosfera AI agent integration with Goose CLI
• AWS Integration: Complete AWS SDK support with CodeArtifact authentication
• Code Server: Includes VS Code server for in-browser development
• Custom Configuration: Optimized environment variables and configurations for Jupyter and Orchest

⁠How to Use It

Add dadosfera/base-kernel-py-agent:<version> (e.g., 1.0.0) as a custom image inside a Project's Environment and build it:

⁠Architecture Overview

⁠User Configuration

• Non-root user: jovyan with sudo privileges
• Working directory: /orchest/services/base-images/base-kernel-py-agent
• Home directory: /home/jovyan

⁠Python Environments

1. Base Environment: Default conda environment with user-facing libraries
2. Python 3.11 Environment: Dedicated environment for Enterprise Gateway kernels
3. Isolated Venv: /home/jovyan/venv for Orchest dependencies

⁠Included Tools and Libraries

⁠System Dependencies

• Development Tools: cmake, curl, git, openssh-server
• Database Support: default-libmysqlclient-dev, libkrb5-dev
• UI Libraries: libxcb1 for graphical applications

⁠Python Package Managers

• uv: Fast Python package installer
• pip: Traditional Python package manager
• mamba: Fast conda package manager

⁠Core Python Libraries

• AWS Integration: boto3, awscli
• Data Processing: numpy<2, pandas, fastparquet
• Data Warehousing: snowflake-snowpark-python
• Web Applications: streamlit
• Utilities: chardet, requests, anybase32, pyopenssl

⁠Dadosfera-Specific Libraries

• dadosfera==1.8.0b6: Core Dadosfera SDK
• dadosfera_logs==1.0.3: Logging utilities

⁠Enterprise Gateway Dependencies

• jupyter_client<7: Jupyter client library
• ipykernel, ipython: Kernel and IPython support
• pycryptodome: Cryptographic library
• cffi, future: Additional dependencies

⁠AI and Development Tools

• Goose CLI: AI-powered debugging and configuration tool
• Code Server: VS Code in the browser
• Dadosfera AI Agent: Custom AI wrapper for enhanced functionality

⁠Environment Variables

⁠Build Instructions

⁠Prerequisites

• Docker with BuildKit support
• AWS credentials as Docker secrets:
  • aws_access_key_id
  • aws_secret_access_key

⁠Build Command

docker build \
  --secret id=aws_access_key_id,src=path/to/aws_access_key_id \
  --secret id=aws_secret_access_key,src=path/to/aws_secret_access_key \
  --build-arg ORCHEST_VERSION=<version> \
  -t dadosfera-base-kernel-py-agent:<tag> .

⁠File Structure

The image includes several custom components:

⁠AI Agent Wrapper

• dadosfera-ai.py: Main AI agent script
• dadosfera-ai.sh: Shell wrapper for AI agent
• setup_venv.sh: Virtual environment setup script

⁠Orchest Integration

• bootscript.sh: Container startup script
• requirements.txt: Orchest dependencies
• requirements-user.txt: User-facing dependencies
• Orchest SDK and library files

⁠Configuration Files

• .orchestrc: Shell configuration for Orchest environment
• Custom sudoers configuration for jovyan user

⁠Usage Scenarios

This image is designed for:

1. Data Science Workflows: Full-featured environment for data analysis and ML
2. Orchest Pipelines: Native support for Orchest workflow orchestration
3. AWS Data Processing: Seamless integration with AWS services
4. AI-Assisted Development: Built-in AI agent for enhanced productivity
5. Collaborative Development: Code server for browser-based development

⁠Customization

⁠Adding Python Dependencies

Modify the requirements files:

• requirements-user.txt: User-facing libraries
• requirements.txt: System/Orchest dependencies

⁠Custom Startup Behavior

Edit bootscript.sh to customize container initialization.

⁠AI Agent Configuration

Modify the AI agent wrapper scripts in the ai-agent-wrapper directory.

⁠MCPS Configuration

The dadosfera-ai Model Context Protocols (MCPS) are located at /usr/local/bin/mcps/ within the container. These scripts provide pre-configured prompts for the AI agent to handle common tasks. You can customize or add new MCPS files to enhance the AI capabilities.

⁠Available MCP Extensions

The container includes the following MCP (Model Context Protocol) extensions:

1. Orchest MCP (orchest-mcp):

   • Purpose: Integration with Orchest workflow orchestration
   • Location: /usr/local/bin/mcps/dadosfera-orchest-mcp
   • Required Environment Variables:
     • ORCHEST_BASE_URL: Base URL for your Orchest instance
     • ORCHEST_USER: Username for Orchest authentication
     • ORCHEST_PASSWORD: Password for Orchest authentication

2. Dadosfera MCP Server (dadosfera-mcp-server):

   • Purpose: Integration with Dadosfera platform services
   • Location: /usr/local/bin/mcps/dadosfera-mcp-server/dadosfera-mcp
   • Required Environment Variables:
     • DADOSFERA_USERNAME: Your Dadosfera username
     • DADOSFERA_PASSWORD: Your Dadosfera password
     • DADOSFERA_METABASE_USERNAME: Metabase username for BI integration
     • DADOSFERA_METABASE_PASSWORD: Metabase password for BI integration
     • DADOSFERA_CUSTOMER_NAME: Your customer/organization name
     • MCP_SERVER_PREFIX: Server prefix for MCP communication
     • MCP_SERVER_PASSWORD: Password for MCP server authentication

3. Built-in Extensions:

   • Computer Controller: System interaction capabilities
   • Developer: Development workflow assistance
   • Memory: Conversation memory management
   • Fetch: HTTP request capabilities
   • Crawler: Web scraping with Puppeteer

⁠MCP Configuration Example

To configure the MCP extensions, set these environment variables in your shell configuration:

# Add to ~/.bashrc or ~/.orchestrc
export ORCHEST_BASE_URL="https://your-orchest-instance.com"
export ORCHEST_USER="your-username"
export ORCHEST_PASSWORD="your-password"

export DADOSFERA_USERNAME="your-dadosfera-username"
export DADOSFERA_PASSWORD="your-dadosfera-password"
export DADOSFERA_METABASE_USERNAME="your-metabase-username"
export DADOSFERA_METABASE_PASSWORD="your-metabase-password"
export DADOSFERA_CUSTOMER_NAME="your-organization"
export MCP_SERVER_PREFIX="your-prefix"
export MCP_SERVER_PASSWORD="your-mcp-password"

⁠Testing MCP Configuration

To verify your MCP configuration is working:

# Test Goose CLI with MCP extensions
dadosfera-ai session start

# Check available tools
dadosfera-ai tools list

# Test specific MCP functionality
dadosfera-ai session "List available Orchest pipelines"
dadosfera-ai session "Show Dadosfera project status"

⁠Docker Image

The final image is pushed to the Docker registry with the tag dadosfera/base-kernel-py-agent:<tag>. It includes all the dependencies and is ready to use in your projects.

⁠License

This project is licensed under the terms of the MIT license. See the LICENSE⁠ file for details.

⁠Acknowledgements

• Thanks to the Jupyter and Conda communities for their excellent base images and package management systems.
• Inspired by the need for a robust, flexible data science environment in the cloud.