farzadras/gpdb-analytics repository overview

Original docker image: hdlee2u/gpdb-analytics

⁠How to run the docker image along with a Jupyter notebook in a linux VM on Azure

The following instructions have been tested using a "Ubuntu 22.04.3 LTS x64 Gen2" virtual machine (region: East US, size: Standard_D2s_v3) on Azure.

1. After deploying the virtual machine (VM), make sure to click on the VM link in the Azure portal and add an inbound port rule for ports 8888 and 5432. Note that the database port 5432 will be exposed this way, but this tutorial can live with that.

2. Connect to the virtual machine using a terminal window and run the following commands.

   $ sudo apt update -y
   $ sudo apt install python3-psycopg2 -y
   

   Copy

3. Install Docker in the virtual machine if needed (either run sudo apt install docker.io or see https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04⁠)

4. Download Classification on Greenplum Using MADlib.ipynb into the virtual machine (using the command below).

   $ wget https://github.com/farzadrastegar/assets/raw/master/Greenplum-ApacheMADlib/notebooks/Classification%20on%20Greenplum%20Using%20MADlib.ipynb
   

   Copy

5. Pull the docker image and run a container for a Greenplum database with MADlib functionality.

   $ sudo docker run -i -d -p 5432:5432 -p 28080:28080 --name gpdb-ds --hostname mdw farzadras/gpdb-analytics:latest /usr/sbin/sshd -D
   

   Copy

6. Connect to the container and change the password of the role gpadmin in the database to something else through the command below (see the next section under the title "To Start Greenplum Database and Use psql" for how to connect to the container).

   ALTER USER gpadmin WITH PASSWORD 'pivotal';
   

   Copy

   

7. Execute the command below to run a minimal Jupyter notebook on port 8888 in the virtual machine.

   $ sudo docker run -d -p "8888:8888" --name jupyter-notebook --net=host -e GRANT_SUDO=yes --user root -v "${PWD}:/home/jovyan/work" jupyter/minimal-notebook
   

   Copy

8. The Jupyter notebook is running now. However, we need a password to connect to it. The password has already been provided to us in the logs of the previous command. To retrieve the password, run the following command.

   $ sudo docker logs jupyter-notebook 2>&1 | grep "token"
   

   Copy

   The image below shows how to grab the password from the output of the command above.

9. Now, connect to the Jupyter notebook using the IP of the virtual machine, e.g., http://10.20.30.40:8888/⁠. (Reminder for mentioning the port number, i.e., 8888, after the IP of the virtual machine.)

10. Use the token extracted before to login (see the image below).

11. Double click on the work directory and then double click on the Jupyter notebook downloaded previously to open.

12. The Jupyter notebook is ready for use now. Make sure to change the IP address and password before connecting to the database (the cell containing this command: postgresql://gpadmin:[email protected]:5432/gpadmin).

    See this Youtube video for further explanations: https://www.youtube.com/watch?v=ht9y50oZUEU⁠

⁠Docker Image Specifications

source: https://hub.docker.com/r/hdlee2u/gpdb-analytics⁠

CentOS : release 7.4

Greenplum Database : ver 5.10.2

MADlib : ver 1.15

PL/R : 2.3.2

DataScienceR : 1.0.1

DataSciencePython : 1.1.1

(1) Docker Image Pull

$ docker pull farzadras/gpdb-analytics
$ docker images

(2) Docker Image Run(port 5432) -> Docker Container Creation

$ docker run -i -d -p 5432:5432 -p 28080:28080 --name gpdb-ds --hostname mdw hdlee2u/gpdb-analytics /usr/sbin/sshd -D
$ docker ps -a

(3) To Start Greenplum Database and Use psql

$ docker exec -it gpdb-ds /bin/bash
[root@mdw /]# su - gpadmin
[gpadmin@mdw ~]$ gpstart -a
[gpadmin@mdw ~]$ psql
psql (8.3.23)
Type "help" for help.

gpadmin=# \dn
List of schemas
Name | Owner
--------------------+---------
gp_toolkit | gpadmin
information_schema | gpadmin
madlib | gpadmin
pg_aoseg | gpadmin
pg_bitmapindex | gpadmin
pg_catalog | gpadmin
pg_toast | gpadmin
public | gpadmin
(8 rows)

gpadmin=# \q

(4) PGAdmin IV configuration for SQL query

Host : localhost
Port : 5432
Maintenance DB : gpadmin
Username : gpadmin
Password : pivotal
Group: Servers
Ternel port: 22

PGAdmin 4 download: https://www.pgadmin.org/download/⁠

(5) Docker Container Stop, Restart

$ docker stop gpdb-ds
$ docker start gpdb-ds
$ docker ps

(6) In case of 'Error response from daemon: driver failed programming external connectivity on endpoint gpdb-ds : Error starting userland proxy: Bind for 0.0.0.0:5432 failed: port is already allocated

• Check a running port

  $ sudo lsof -i :5432
  

  Copy

• Kill a port which is running

  $ sudo kill -15 93
  

  Copy

• For more information on Greenplum Database https://greenplum.org/⁠ https://network.pivotal.io/⁠ http://rfriend.tistory.com/377⁠

For more information

• For more information on Apache MADlib (Big Data Machine Learning in SQL) http://madlib.apache.org/⁠

• For more information on Greenplum PL/R Language Extension https://gpdb.docs.pivotal.io/590/ref_guide/extensions/pl_r.html⁠

• For more information on Greenplum PL/Python Language Extension https://gpdb.docs.pivotal.io/590/ref_guide/extensions/pl_python.html⁠

• For more information on Docker Install, GPDB start, PGAdmin setup, Jupyter Notebook setup http://rfriend.tistory.com/379⁠

• For more information on 'Error starting userland proxy: Bind for 0.0.0.0:5432 failed: port is already allocated' http://rfriend.tistory.com/396⁠