Sign inSign up

jorgecardona/jupyterlabmultilanguagespython3117

By jorgecardona

•Updated over 2 years ago

Jupyter Git Python Java Scala R Node Kotlin Py-Spark Beam Scikit Dask Kafka DBT Airflow PyTest Panel

Image
0

1.5K

jorgecardona/jupyterlabmultilanguagespython3117 repository overview

⁠Available Kernels and Languages

AplicationPort
JupyterLab8888
Apache Spark UI4040
Panel5006
Node.js3000
dbt8080
Airflow8081
Kafka9091-9094

⁠EXAMPLES BY LANGUAGE

LanguageExample
Javapublic class Main {
    public static void greeting() {
        System.out.println("Hello, Java!");
    }
}
Main.greeting();
Node.jsconst http = require('http');

// Create the server
const server = http.createServer((req, res) => {
    // Configure the server response
    res.statusCode = 200;
    res.setHeader('Content-Type', 'text/plain');
    res.end('Hello, from JupyterLab Container!');
});

// Define the port on which the server will run
const port = 3000;

// Start the server
server.listen(port, () => {
    console.log('Node.js server is running at http://localhost⁠:' + port + '/');
});
Kotlinfun main() {
    // Print "Hello, Kotlin!"
    println("Hello, Kotlin!")

    // Define a function to calculate the square of a number
    fun calculateSquare(number: Int): Int {
        return number * number
    }

    // Use the function to calculate the square of 5
    val result = calculateSquare(5)
    println("Square of 5: $result")
}
main()
R# Print "Hello, R!"
cat("Hello, R!\n")

# Define a function to calculate the square of a number
calculateSquare <- function(number) {
    return (number * number)
}

# Use the function to calculate the square of 5
result <- calculateSquare(5)
cat("Square of 5: ", result, "\n")
Scala// Print "Hello, Scala!"
println("Hello, Scala!")

// Define a function to calculate the square of a number
def calculateSquare(number: Int): Int = {
    number * number
}

// Use the function to calculate the square of 5
val result = calculateSquare(5)

// Print the result
println("The square of 5 is: " + result)

⁠Packages Installed

PackageVersion
jupyterlab4.0.10
jupyterlab-git0.50.0
pyspark3.4.1
pandas1.5.3
dask2024.1.0
apache-beam[interactive]2.48.0
panel1.3.6
Faker22.0.0
itables1.6.3
dbt-core1.7.4
apache-airflow2.8.0
mysql-connector-python8.2.0
psycopg22.9.9
pymongo4.6.1

⁠EXAMPLES OF INSTALLED PACKAGES

⁠You can use the following examples for testing, just copy and paste the following code to test each package.

PackageDescriptionAdvantagesDisadvantagesExample
itablestabular data manipulation in Python.SQL-like syntax for querying.May be less efficient for advanced operations compared to specialized libraries.# to enable itables
from itables import init_notebook_mode
init_notebook_mode(all_interactive=True)
pysparkdistributed processing with Apache Spark- Distributed and scalable processing- Complex configuration and managementfrom pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder.appName("JorgeCardonaSpark").getOrCreate()

# Perform a simple DataFrame operation
data = [('Nathalie', 0), ('Ana', 3), ('Diana', 7), ('Lucia', 10), ('Tatiana', 13), ('Angela', 17), ('Cecilia', 25), ('Alice', 31), ('Kristin', 35), ('Carolina', 37), ('Lina', 39), ('Marcela', 40), ('Maria', 42)]

# Create a Dataframe
df = spark.createDataFrame(data, ["Name", "Age"])
df.show()
spark.stop()
pandasdata manipulation and analysis- Efficient data manipulation and analysis functions- Limitations in handling large volumes of dataimport pandas as pd

data = {
    'Name': ["Nathalie", "Ana", "Diana", "Lucia", "Tatiana", "Angela", "Cecilia", "Alice", "Kristin", "Carolina", "Lina", "Marcela", "Maria"],
    'Age': [0, 3, 7, 10, 13, 17, 25, 31, 35, 37, 39, 40, 42]
}
df = pd.DataFrame(data)
df
apache-beamProgramming model for data processing- High-level abstraction for data processing- Requires knowledge of parallel programmingimport apache_beam as beam

def regular_case_function(element):
    return element.lower()

def to_uppercase_function(element):
    return element.upper()

def calculate_length_function(element):
    return len(element)

def calculate_square_function(element):
    return element ** 2

# Create a pipeline
with beam.Pipeline() as pipeline:
    # Prepare a list of names to be processed
    names_list = ["Nathalie", "Ana", "Diana", "Lucia", "Tatiana", "Angela", "Cecilia", "Alice", "Kristin", "Carolina", "Lina", "Marcela", "Maria"]

    # Create a PCollection with the given data
    data = pipeline | beam.Create(names_list)

    # Apply transformation functions to the data
    regular_case_data = data | beam.Map(regular_case_function) # Transform to lowercase
    uppercase_data = data | beam.Map(to_uppercase_function) # Transform to uppercase
    length_data = data | beam.Map(calculate_length_function) # Apply transformation to calculate the length of each name
    square_data = length_data | beam.Map(calculate_square_function) # Apply transformation to calculate the square

    # Print the results of each transformation
    length_data | "Show_Length" >> beam.Map(print) # Print length results
    regular_case_data | "Show_Lowercase" >> beam.Map(print) # Print lowercase results
    uppercase_data | "Show_Uppercase" >> beam.Map(print) # Print uppercase results
    square_data | "Show_Square" >> beam.Map(print) # Print square results
    combined_data = (length_data, regular_case_data, uppercase_data, square_data) | beam.Flatten()
    combined_data | "Show_All" >> beam.Map(print)
FakerGenerating simulated data- Easy generation of simulated data- Not suitable for production environmentsfrom faker import Faker
fake = Faker()
name = fake.name()
print(name)
PanelCreating interactive dashboards and apps- Powerful dashboard and app creation capabilities- Requires learning the Panel libraryimport panel as pn

def model(n=5):
    return "⭐"*n

pn.extension()

slider = pn.widgets.IntSlider(value=5, start=1, end=5)

interactive_model = pn.bind(model, n=slider)

layout = pn.Column(slider, interactive_model)

app = pn.serve(layout, port=5006, show=True)

#app.stop()

# ✨ Panel UI APP http://localhost:5006 CLICK HERE ⁠✨

⁠⚠️ For this image, it is not necessary to use the token to access the notebooks ⚠️

⁠How to run the image.

⁠TEMPORAL CONTAINER

⁠🔥If you want to just test the image and do not keep the container when you finish running the container use the next command:

⁠ docker run --name jorgecardona-labmultilanguage --rm -p 8888:8888 -p 4040:4040 -p 5006:5006 -p 3000:3000 -p 8080:8080 -p 8081:8081 -p 9091:9091 -p 9092:9092 -p 9093:9093 -p 9094:9094 jorgecardona/jupyterlabmultilanguagespython3117:latest

⁠PERSISTENT CONTAINER

⁠💦If you want to keep the container, save the notebooks, and continue working on this container use the next command:💦

⁠ docker run --name jorgecardona-labmultilanguage -p 8888:8888 -p 4040:4040 -p 5006:5006 -p 3000:3000 -p 8080:8080 -p 8081:8081 -p 9091:9091 -p 9092:9092 -p 9093:9093 -p 9094:9094 jorgecardona/jupyterlabmultilanguagespython3117:latest

⁠ACCESS TO JUPYTER LAB AND SPARK UI

⁠🐱 access to JUPYTERLAB http://localhost:8888 CLICK HERE ⁠ 🐱

⁠🐶access to sparkUI with pySpark http://localhost:4040 CLICK HERE ⁠🐶

⁠🌀To run Spark for Scala, on the terminal execute spark-shell command to start it.🌀

⁠🐍access to sparkUI Running directly Apache Spark http://localhost:4040 CLICK HERE ⁠🐍

⁠ADDED SUPPORT TO CONNECT SPARK FOR EXTRACTION DATA FROM MySQL, MongoDB, AND PostgreSQL

⁠PING to databases MySQL, MongoDB, PostgreSQL

⁠USE TEMPORAL DOCKER IMAGES FOR TESTING, delete --rm on docker command to do persistent databases

⁠DOCKER FOR PostgreSQL

⁠docker run --name jorgecardona-postgres --rm -e POSTGRES_DB=spark -e POSTGRES_PASSWORD=12345678 -e POSTGRES_USER=admin -d -p 5432:5432 postgres:15.4

⁠DOCKER FOR MySQL

⁠docker run --name jorgecardona-mysql --rm -e MYSQL_DATABASE=spark -e MYSQL_PASSWORD=12345678 -e MYSQL_USER=admin -e MYSQL_ROOT_PASSWORD=root -d -p 3306:3306 mysql:8.1.0

⁠DOCKER FOR MongoDB

⁠docker run --name jorgecardona-mongodb --rm -d -p 27017:27017 -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=12345678 mongodb/mongodb-community-server:6.0.7-ubuntu2204-20230812T065949Z

⁠STRING CONNECTION FOR SPECIFIC MONGO DB HOST

⁠mongodb://admin:12345678@localhost:27017

⁠STRING CONNECTION FOR LOCAL DOCKER MONGO DB

⁠mongodb://admin:[email protected]:27017

⁠CODE FOR TESTING CONNECTION TO DATABASES

def test_mongo_connection(host, port, database, collection, user=None, password=None):
    """
    host = "host.docker.internal"  # Replace with the IP address or hostname of your MongoDB server
    port = 27017
    database = "spark"
    collection = "users"
    user = "admin"  # Username (optional, if MongoDB is configured with authentication)
    password = "12345678"  # Password (optional, if MongoDB is configured with authentication)
    # If you want to use authentication, you need to provide credentials
    # mongodb://admin:12345678@localhost:27017
    # test the connection
    test_mongo_connection(host, port, database, collection)
    test_mongo_connection(host, port, database, collection, user, password)
    """
    from pymongo import MongoClient
    try:
        client = MongoClient(host, port, username=user, password=password)
        if not user or not password:
            client = MongoClient(host, port)
        db = client[database]
        # You can use any query here; for example, count_documents({})
        collection_loaded = db[collection]
        result = collection_loaded.find_one()
        if result:
            print("Connection successful. MongoDB server is accessible.", result)
        else:
            print("Connection successful, but no data was found in the database.")
        client.close()
    except Exception as e:
        print(f"Error connecting to the database: {e}")
                
def test_postgres_connection(host, port, database, user, password):
    """
    host = "host.docker.internal"  # Replace with the IP address or hostname of your PostgreSQL server
    port = 5432
    database = "spark"
    user = "admin"
    password = "12345678"
    # test the connection
    test_postgres_connection(host, port, database, user, password)
    """
    import psycopg2
    try:
        connection = psycopg2.connect(host=host, port=port, database=database, user=user, password=password)
        connection.close()
        print("Connection successful. PostgreSQL server is accessible.")
    except Exception as e:
        print(f"Error connecting to the database: {e}")
                
def test_mysql_connection(host, port, database, user, password):
    """
    host = "host.docker.internal"  # Replace with the IP address or hostname of your MySQL server
    port = 3306  # Default port for MySQL
    database = "spark"  # Name of the database you want to connect to
    user = "admin"  # Username
    password = "12345678"  # Password
    # test the connection
    test_mysql_connection(host, port, database, user, password)
    """    
    import mysql.connector
    try:
        connection = mysql.connector.connect(host=host, port=port, database=database, user=user, password=password)
        connection.close()
        print("Connection successful. MySQL server is accessible.")
    except Exception as e:
        print(f"Error connecting to the database: {e}")

⁠DATABASES CONFIGURATION USING SPARK SESSION

def get_database_configuration(database_type = 'mysql', host = None, port = None, database = None, table = None, user = None, password = None, input_collection = None, output_collection = None):
    databases = {
        'mongodb': {
            'app_name': 'MongoDB_Connector',
            'format_type':'mongodb',
            'host': host if database_type == 'mongodb' and host else 'host.docker.internal',
            'port': port if database_type == 'mongodb' and port else 27017,
            'user': user if database_type == 'mongodb' and user else 'admin',
            'password': password if database_type == 'mongodb' and password else '12345678',
            'database': database if database_type == 'mongodb' and database else 'spark',
            'input_collection':  input_collection if database_type == 'mongodb' and table else 'users',
            'output_collection': output_collection if database_type == 'mongodb' and table else 'users',
            'driver': 'com.mongodb.spark.sql.DefaultSource',
            'url': f"mongodb://{user}:{password}@{host}:{port}" if database_type == 'mongodb' and host and port else 'mongodb://admin:[email protected]:27017'
            },
        'postgres': {
            'app_name': 'PostgreSQL_Connector',
            'format_type':'jdbc',
            'host': host if database_type == 'postgres' and host else 'host.docker.internal',
            'port': port if database_type == 'postgres' and port else 5432,
            'user': user if database_type == 'postgres' and user else 'admin',
            'password': password if database_type == 'postgres' and password else '12345678',
            'database': database if database_type == 'postgres' and database else 'spark',
            'table': table if database_type == 'postgres' and table else 'users',
            'schema': 'public',
            'spark_jars': '/usr/local/spark/jars/postgresql-42.7.1.jar',
            'driver': 'org.postgresql.Driver',
            'url': f"jdbc:postgresql://{host}:{port}/{database}" if database_type == 'postgres' and host and port else 'jdbc:postgresql://host.docker.internal:5432/spark',
            'properties': {
                'user': user if database_type == 'postgres' and user else 'admin',
                'password': password if database_type == 'postgres' and password else '12345678',
                'driver': 'org.postgresql.Driver'
                            }
            },
        'mysql': {
            'app_name': 'MySQL_Connector',
            'format_type':'jdbc',
            'host': host if database_type == 'mysql' and host else 'host.docker.internal',
            'port': port if database_type == 'mysql' and port else 3306,
            'user': user if database_type == 'mysql' and user else 'admin',
            'password': password if database_type == 'mysql' and password else '12345678',
            'database': database if database_type == 'mysql' and database else 'spark',
            'table': table if database_type == 'mysql' and table else 'users',
            'spark_jars': '/usr/local/spark/jars/mysql-connector-j-8.2.0.jar',
            'driver': 'com.mysql.cj.jdbc.Driver',
            'url': f"jdbc:mysql://{host}:{port}/{database}" if database_type == 'mysql' and host and port else 'jdbc:mysql://host.docker.internal:3306/spark',
            'properties': { 
                            'user': user if database_type == 'mysql' and user else 'admin', 
                            'password': password if database_type == 'mysql' and password else '12345678', 
                            'driver': 'com.mysql.cj.jdbc.Driver'
            }
                            }
    }
    return databases.get(database_type.lower(), databases.get('mysql'))

⁠INSERT DATA - TEST DATABASES

def generate_sample_data():
    data = [(1, "Ana"), (2, "Cecilia"), (3, "Nathalie"), (4, "Diana"), (5, "Gabriela"), (6, "Angela"), (7, "Tatiana"), (8, "Lucia"), (9, "Maria")]
    columns = ["Id", "Name"]
    return data, columns

def insert_data_to_database(database_configuration, database_type=None):
    from pyspark.sql import SparkSession
    from pyspark.sql.functions import monotonically_increasing_id
    
    app_name = database_configuration.get('app_name')
    format_type = database_configuration.get('format_type')
    database = database_configuration.get('database')
    user = database_configuration.get('user')
    password = database_configuration.get('password')
    driver = database_configuration.get('driver')
    url = database_configuration.get('url')
    spark_session = SparkSession.builder.master('local').appName(app_name)
    
    data, columns = generate_sample_data()

    try:
        message = f'Records Inserted Successfully in {app_name}'
        if database_type == 'mongodb':
            collection = database_configuration.get('output_collection')
            spark_session = spark_session.getOrCreate()
            sampleDF = spark_session.createDataFrame(data, columns)
            sampleDF_with_id = sampleDF.withColumn("id", monotonically_increasing_id()) # add column 'id' to DataFrame

            sampleDF_with_id.write.format("mongodb") \
            .option("connection.uri", url).option("database", database) \
            .option("collection", collection).mode("append").save()
        else: 
            dbtable = database_configuration.get('table')
            spark_jars = database_configuration.get('spark_jars')
            spark_session = spark_session.config("spark.jars", spark_jars)
            spark_session = spark_session.config("spark.jars", spark_jars).getOrCreate()
            sampleDF = spark_session.createDataFrame(data, columns)
            sampleDF.write \
                .format(format_type).option("driver", driver) \
                .option("url", url).option("dbtable", dbtable) \
                .option("user", user).option("password", password) \
                .mode("ignore").mode("append").save()
    except Exception as e:
        message = f"Error inserting data: {str(e)}"
    finally:
        spark_session.stop() # stop Spark session
        return message

⁠MYSQL, GET DATABASE CONFIGURATION - INSERT DATA INTO DATABASE AND CREATES THE TABLE

mysql_configuration = get_database_configuration(database_type = 'mysql')
insert_data_to_database(mysql_configuration)

⁠POSTGRESQL, GET DATABASE CONFIGURATION - INSERT DATA INTO DATABASE AND CREATES THE TABLE

postgres_configuration = get_database_configuration(database_type = 'postgres')
insert_data_to_database(postgres_configuration)

⁠MONGODB, , GET DATABASE CONFIGURATION - INSERT DATA INTO DATABASE AND CREATES THE COLLECTION

mongodb_configuration = get_database_configuration(database_type = 'mongodb')
insert_data_to_database(database_configuration=mongodb_configuration, database_type = 'mongodb')

⁠READ DATA - TEST DATABASES

def read_data_from_database(database_type='mysql', host=None, port=None, database=None, table=None, user=None, password=None, input_collection=None, output_collection=None):
    database_configuration = get_database_configuration(database_type=database_type, host=host, port=port, database=database, table=table, user=user, password=password, input_collection=input_collection, output_collection=output_collection)
    from pyspark.sql import SparkSession
    spark_session =  SparkSession.builder.master('local').appName(f'Read data from {database_type}').getOrCreate()
    properties = database_configuration.get('properties')
    url = database_configuration.get('url')
    try:
        if database_type == 'mongodb':
            database = database_configuration.get('database')
            collection = database_configuration.get('input_collection')
            result = spark_session.read.format("mongodb").option("connection.uri", url).option("database", database).option("collection", collection).load()
        else:
            table = database_configuration.get('table')
            if database_configuration.get('schema'):
                table = f"{database_configuration.get('schema')}.{database_configuration.get('table')}"
            result = spark_session.read.jdbc(url=url, table=table, properties=properties)
    except Exception as e:
        print(f"Error reading data: {str(e)}")
        result = None  # Another action you may want to take in case of an exception
    finally:
        result.printSchema()  # Print schema
        result.show()  # Show rows
        df = result.toPandas() # converts to pandas
        spark_session.stop()
        return df

⁠MYSQL

read_data_from_database(database_type = 'mysql')

⁠POSTGRESQL

read_data_from_database(database_type = 'postgres')

⁠MONGODB

read_data_from_database(database_type = 'mongodb')

⁠DBT EXAMPLE CONFIGURATION

dbt init dbt_poc
Enter a number: 1
host (hostname for the instance): host.docker.internal
port [5432]: 5432
user (dev username): admin
pass (dev password): 12345678
dbname (default database that dbt will build objects in): spark
schema (default schema that dbt will build objects in): public
threads (1 or more) [1]: 1

⁠DBT TESTING

cd  dbt_poc
dbt debug # test connection
dbt run # creates the tables
dbt docs generate # generates documentation
dbt docs serve

⁠🌀 dbt_poc DOCUMENTATION CLICK HERE⁠ 🌀

⁠AIRFLOW TESTING

airflow scheduler
airflow webserver -p 8081
user : admin
pass: 12345678

⁠💠 Airflow Interface CLICK HERE⁠💠

⁠KAFKA TESTING, RUN ZOOKEEPER, START BROKERS, LIST BROKERS, KAFKA SPARK STREAMING AND BATCH

⁠🐳 Spark-Kafka Notebook GUIDE CLICK HERE ⁠🐳

⁠stop the container CONTAINER_ID or NAME

⁠docker stop jorgecardona-labmultilanguage jorgecardona-postgres jorgecardona-mysql jorgecardona-mongodb

⁠re start stopped CONTAINER_ID or NAME

⁠docker start -i jorgecardona-labmultilanguage

⁠remove container CONTAINER_ID or NAME

⁠docker rm jorgecardona-labmultilanguage

⁠🐳 Original Dockerfile for this image CLICK HERE ⁠🐳

Tag summary

Content type

Image

Digest

sha256:c25f9241d…

Size

3.1 GB

Last updated

over 2 years ago

docker pull jorgecardona/jupyterlabmultilanguagespython3117