shrysr/shiny
Readme on Readthedocs:
Docker image Cloud Build Status. Note: Sometimes images are built locally and pushed dockerhub
Quick Commands:
Docker Images:
docker pull shrysr/shiny:v2
docker pull shrysr/rstudio:v2
Docker compose to launch services:
docker compose up -d rstudio
docker compose up -d shiny
# To launch both : docker compose up -d
The starting point of this project was Matt Dancho's shinyauth docker file, which then expanded into adapting the well designed Matrix DS tools for my purpose. Their stack of docker based tools is replicated here with my customization which adds libraries and other functionality.
The goal is to develop a workflow based on Docker (and other tools) to create a reproducible, standard, consistent environment to run a variety of datascience projects, with different development and production environments. In particular, I want to be able to develop and deploy dashboards like shiny or streamlit.io quickly and with ease.
This Readme consists of the entire documentation and source code, maintained in a single Org mode document which is used to regenerate and manage the entire source code and the exports to markdown and rst. i.e Literate Programming in all it's splendor and warts.
Why not just use the MatrixDS stack directly and add the missing packages as layers?
In a sense, that is what I did, except that I constructed my own base image instead of relying on modifying a MatrixDS image. I also wanted to build these images by hand as my set of tools, even if the tools were largely similar to the MatrixDS stack. From whatever I've learned of Docker - the MatrixDS stack is quite efficient and the cascading + common dependency layer makes sense to use. There may be other methods, but this certainly appeared technically sensible.
The main containers to be aware of, and also hosted on dockerhub are :
Note the tags to be used with the images.
The rbase image is built on the first asmith image. The RStudio and Shiny images are based of a common rbase dependency environment. However, additional packages can be specified for these, and it is not necessary to rebuild the rbase layer each time.
The easiest way to launch is to use the docker-compose file.
Starting an rstudio service is as simple as:
docker-compose up rstudio
Replace rstudio
with shiny
above to launch the shiny server.
Note that the docker-compose setup involves using a fully specified project path. This is set to a generic path and the system has been tested on a Mac OS and several compute instances of Linux running Debian and Ubuntu.
The purpose of the docker container to is make the environment variables and other settings like volume names and ports to be conveniently configured.
There are 2 services : rstudio server and a shiny server. The default up command will launch both of them. Individual services can be launched using their names.
It would b a good practice to name the containers based on the project so as to be able to distinguish different containers.
version: "2"
services:
rstudio:
image: shrysr/rstudio:v2
container_name: rstudio_s
environment:
- PASSWORD=shreyas
- DISABLE_AUTH=false
- WORKDIR="/home/rstudio/"
restart: always
volumes:
- /Users/shrysr/docker-testing-sr/:/home/rstudio/
ports:
- "8787:8787"
shiny:
image: shrysr/shiny:v2
container_name: shiny
ports:
- "3838:3838"
restart: always
volumes:
- /Users/shrysr/docker-testing-sr/shiny-server/:/srv/shiny-server/
This travis file uses the docker service specification to start the docker service. This can be viewed by the systemctl start
command in the logs. The build is designed in stages, due to the cascading image dependency. However, the rstudio and shiny images can be constructed in parallel and actually do not take long.
Note that the entire build does not complete on Travis at the moment because of limitations in the free tier of computing time. However, the builds do complete on Github Acions described subsequently.
services:
- docker
jobs:
include:
- stage: asmith
script: docker build asmith/. -t shrysr/asmith:v1
- stage: rbase
script: docker build rbase/. -t shrysr/rbase:v2
- stage: rstudio-and-shiny
script: docker build rstudio/. -t shrysr/rstudio:v2
script: docker build shiny/. -t shrysr/shiny:v2
name: Docker Image CI
on:
push:
paths:
- '!/docs/*'
- '!Readme.*'
- '!*.md'
- '!*.org'
jobs:
build:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v1
- name: Build Asmith
run: docker build asmith/. --file asmith/Dockerfile --tag my-image-name:$(date +%s)
name: Docker Image CI
on:
push:
paths:
- '!/docs/*'
- '!Readme.*'
- '!*.md'
- '!*.org'
jobs:
build:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v1
- name: Build rbase
run: docker build rbase/. --file rbase/Dockerfile --tag my-image-name:$(date +%s)
name: Docker Image CI
on:
push:
paths:
- '!/docs/*'
- '!Readme.*'
- '!*.md'
- '!*.org'
jobs:
build:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v1
- name: Build rstudio
run: docker build rstudio/. --file rstudio/Dockerfile --tag my-image-name:$(date +%s)
name: Docker Image CI
on:
push:
paths:
- '!/docs/*'
- '!Readme.*'
- '!*.md'
- '!*.org'
jobs:
build:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v1
- name: Build shiny
run: docker build shiny/. --file shiny/Dockerfile --tag my-image-name:$(date +%s)
This is the very first layer. This layer adds several OS packages and starts with a specific version of Ubuntu (v18.04). Currently, it is largely left the same except for adding the package dtrx, which is useful to quickly zip and unzip files.
This layer does not take very long to build, however, if it is - then all the other subsequent layers will probably need to be rebuilt.
FROM ubuntu:18.04
LABEL maintainer="Shreyas Ragavan <sr@eml.cc>" \
version="1.0"
USER root
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update
# Install all basic OS dependencies
RUN apt-get update \
&& apt-get install -yq --no-install-recommends \
apt \
apt-utils \
bash-completion \
build-essential \
byacc \
bzip2 \
ca-certificates \
emacs \
file \
flex \
fonts-dejavu \
fonts-liberation \
fonts-texgyre \
g++ \
gcc \
gettext \
gfortran \
git \
gnupg2 \
gsfonts \
hdf5-tools \
icu-devtools \
jed \
lmodern \
locales \
make \
mesa-common-dev \
nano \
netcat \
openjdk-8-jdk \
pandoc \
software-properties-common \
sudo \
texlive-fonts-extra \
texlive-fonts-recommended \
texlive-generic-recommended \
texlive-latex-base \
texlive-latex-extra \
texlive-xetex \
tzdata \
unzip \
vim \
wget \
zip \
libsodium-dev \
&& echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
&& locale-gen en_US.utf8 \
&& /usr/sbin/update-locale LANG=en_US.UTF-8
# make the "en_US.UTF-8" locale so postgres will be utf-8 enabled by default
ENV LANG=en_US.utf8 \
LC_ALL=en_US.UTF-8 \
TERM=xterm \
APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1
# Install additional libraries
RUN apt-get install -yq --no-install-recommends \
libblas-dev \
libcurl4 \
libcurl4-gnutls-dev \
libgdal-dev \
libglu1-mesa-dev \
libgmp3-dev \
libicu60 \
libjpeg-turbo8 \
libmagick++-dev \
libmariadb-client-lgpl-dev \
libmpfr-dev \
libmpfr-dev \
libncurses5-dev \
libnettle6 \
libnlopt-dev \
libopenblas-dev \
libpango1.0-0 \
libpangocairo-1.0-0 \
libpng16-16 \
libpq-dev \
libsasl2-dev \
libsm6 \
libssl-dev \
libtiff5 \
libtool \
libudunits2-dev \
libxext-dev \
libxml2-dev \
libxrender1 \
zlib1g-dev \
dtrx
# Set timezone noninteractively
RUN ln -fs /usr/share/zoneinfo/US/Pacific /etc/localtime
# Python stuff
RUN apt-get install -y --no-install-recommends \
python-pip \
python-setuptools \
python-wheel \
python-dev \
python3-pip \
python3-setuptools \
python3-wheel \
python3-dev \
&& apt-get clean
#install git, vim
RUN apt-get install -y git \
vim \
curl
#install kaggle cli
RUN pip install kaggle dvc tensorflow keras pandas
#mongo cli
RUN apt-get install -y mongodb-clients
#mysql shell
RUN apt-get install -y mysql-client
#postgre shell
RUN apt-get install -y postgresql-client
# Add Tini
ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]
RUN apt-get clean \
&& rm -rf /var/lib/apt/lists/*
This layer contains all the basic R packages required for datascience and ML. A bunch of packages were added to the already extensive default list of packages in MatrixDS's docker file.
The packages are defined in an R script called packages.R.
This layer takes a tremendously long time to build. A couple of hours on a Macbook Pro 2019, with 6 cores and 32 GB of RAM. One should be careful in assessing whether this layer has to be disturbed. Automated builds on Dockerhub are likely to take even longer.
Note: As such the dockerfile indicates that the packages are called in the last 2 layers only. It may be possible that subsequent image builds do not take as much time as I imagine.
This is a list of the basic packages being installed. These conver many commonly used libraries for data science. This layer will take a Long time to install.
Do not install custom libraries to this layer. Install in the next layer.
#Script for common package installation on MatrixDS docker image
p<-c('nnet','kknn','randomForest','xgboost','tidyverse','plotly','shiny','shinydashboard',
'devtools','FinCal','googleVis','DT', 'kernlab','earth',
'htmlwidgets','rmarkdown','lubridate','leaflet','sparklyr','magrittr','openxlsx',
'packrat','roxygen2','knitr','readr','readxl','stringr','broom','feather',
'forcats','testthat','plumber','RCurl','rvest','mailR','nlme','foreign','lattice',
'expm','Matrix','flexdashboard','caret','mlbench','plotROC','RJDBC','rgdal',
'highcharter','tidyquant','timetk','quantmod','PerformanceAnalytics','scales',
'tidymodels','C50', 'parsnip','rmetalog','reticulate','umap', 'glmnet', 'easypackages', 'drake', 'shinythemes', 'shinyjs', 'recipes', 'rsample', 'rpart.plot', 'remotes', 'DataExplorer', 'inspectdf', 'janitor', 'mongolite', 'jsonlite', 'config' )
install.packages(p,dependencies = TRUE)
Add your custom packages to this layer. In this way, only the additional packages are installed in a new layer.
#Script for common package installation on MatrixDS docker image
PKGS <- c(
"tidyverse", "mapproj", "maps", "genius", "shinycssloaders", "gmailr"
)
install.packages(PKGS, dependencies = TRUE)
# These packages are sometimes not available for the current R version
# , and therefore installed directly from github
devtools::install_github("tidyverse/googlesheets4", dependencies = TRUE)
devtools::install_github("PMassicotte/gtrendsR", dependencies = TRUE)
FROM shrysr/asmith:v1
LABEL maintainer="Shreyas Ragavan <sr@eml.cc>" \
version="1.0"
#install some helper python packages
RUN pip install sympy numpy
# R Repo, see https://cran.r-project.org/bin/linux/ubuntu/README.html
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/' >> /etc/apt/sources.list
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
RUN add-apt-repository ppa:marutter/c2d4u3.5
# R-specific packages
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
r-base \
r-base-core \
r-recommended \
r-base-dev \
r-cran-boot \
r-cran-class \
r-cran-cluster \
r-cran-codetools \
r-cran-foreign \
r-cran-kernsmooth \
r-cran-matrix \
r-cran-rjava \
r-cran-rpart \
r-cran-spatial \
r-cran-survival
COPY packages.R /usr/local/lib/R/packages.R
COPY custom_packages.R /usr/local/lib/R/custom_packages.R
# Install Basic R packages for datascience and ML
RUN R CMD javareconf && \
Rscript /usr/local/lib/R/packages.R
# Though this has been installed upstream, apprently it has to be setup again.
RUN apt-get update \
&& apt-get install -y --no-install-recommends libsodium-dev
# Install custom set of R packages. This is on a separate layer for efficient image construction
RUN Rscript /usr/local/lib/R/custom_packages.R
*
Overview of the process:
Suppose you have a project folder within which related scripts, shiny apps, etc live. This directory is mounted as a volume to the docker container. The docker container will check for the presence of a folder called shiny-server
and if not available, will create it. Even if the folder is available, the contents of test_apps will be copied into the image.
Into the shiny-server
folder, the test_apps folder containing shiny apps for testing are copied.
R_LIBS=/usr/local/lib/R/site-library:/usr/local/lib/R/library:/usr/lib/R/library:/srv/R/library
.libPaths("/srv/R/library/")
# Things you might want to change
# options(papersize="a4")
# options(editor="notepad")
# options(pager="internal")
# R interactive prompt
# options(prompt="> ")
# options(continue="+ ")
# to prefer Compiled HTML
help options(chmhelp=TRUE)
# to prefer HTML help
# options(htmlhelp=TRUE)
# General options
options(tab.width = 4)
options(width = 130)
options(graphics.record=TRUE)
.First <- function(){
library(Hmisc)
library(R2HTML)
cat("\nWelcome at", date(), "\n")
}
.Last <- function(){
cat("\nGoodbye at ", date(), "\n")
}
#
# This is a Shiny web application on MatrixDS.
#
# Find out more about building applications with Shiny here:
#
# http://shiny.rstudio.com/
#
##########################################################################################
# This points the Shiny server tool to any libraries installed with RStudio
# that means that any library you install on your RStudio instance in this project,
# will be available to the shiny server
##########################################################################################
.libPaths( c( .libPaths(), "/srv/.R/library") )
##########################################################################################
# Here you can call all the required libraries for your code to run
##########################################################################################
library(shiny)
##########################################################################################
# For deploying tools on MatrixDS, we created this production variable
# when set to true, your shiny app will run on the shiny server tool upon clicking open
# when set to false, your shiny app will run when you hit the "Run App" button on RStudio
##########################################################################################
production <- TRUE
##########################################################################################
# The shiny server tool uses a different absolute path than RStudio.
# this if statement denotes the correct path for the 2 values of the production variable
##########################################################################################
if(production == FALSE) {
#if you using the RStudio tool
shiny_path <- "~/shiny-server/"
home_path <- "~/"
} else {
#if you are using the shiny tool
shiny_path <- "/srv/shiny-server/"
home_path <- "/srv/"
}
##########################################################################################
# To call a file/artifact in your MatrixDS project use the following line of code
# this example uses the function read.csv
# my_csv <- read.csv(paste0(home_path,"file_name.csv"))
##########################################################################################
# Define UI for application that draws a histogram
ui <- fluidPage(
# Application title
titlePanel("Old Faithful Geyser Data"),
# Sidebar with a slider input for number of bins
sidebarLayout(
sidebarPanel(
sliderInput("bins",
"Number of bins:",
min = 1,
max = 50,
value = 30)
),
# Show a plot of the generated distribution
mainPanel(
plotOutput("distPlot")
)
)
)
# Define server logic required to draw a histogram
server <- function(input, output) {
output$distPlot <- renderPlot({
# generate bins based on input$bins from ui.R
x <- faithful[, 2]
bins <- seq(min(x), max(x), length.out = input$bins + 1)
# draw the histogram with the specified number of bins
hist(x, breaks = bins, col = 'darkgray', border = 'white')
})
}
# Run the application
shinyApp(ui = ui, server = server)
This is script to execute or run the shiny server. Apparently, it is necessary to be called via script in this fashion for the process to work, rather than the docker file itself. In a way this helps keeping the code modular. It is generally unlikely any changes would be needed here.
#!/bin/sh
# Make sure the directory for individual app logs exists
mkdir -p /var/log/shiny-server
chown shiny.shiny /var/log/shiny-server
if [ "$APPLICATION_LOGS_TO_STDOUT" = "false" ];
then
exec shiny-server 2>&1
else
# start shiny server in detached mode
exec shiny-server 2>&1 &
# push the "real" application logs to stdout with xtail
exec xtail /var/log/shiny-server/
fi
#Script for common package installation on MatrixDS docker image
p<-c('reticulate')
install.packages(p,dependencies = TRUE)
The folder test_apps will contain shiny apps meant to test functionality. This is copied into the docker image.
Changes: Reduced a step and added the tree package. This makes it easier to troubleshoot.
FROM shrysr/rbase:v2
LABEL maintainer="Shreyas Ragavan <sr@eml.cc>" \
version="2.0"
COPY packages.R /usr/local/lib/R/packages.R
#install R packages
RUN R CMD javareconf && \
Rscript /usr/local/lib/R/packages.R
RUN apt-get update && apt-get install -y \
gdebi-core \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
tree
COPY entrypoint.sh /entrypoint.sh
RUN mkdir -p /root/shiny-server/ \
&& mkdir -p /root/shiny-server/test_shiny/
COPY test_apps/ /root/shiny-server/test_shiny/
# Download and install shiny server
RUN wget --no-verbose https://download3.rstudio.org/ubuntu-14.04/x86_64/VERSION -O "version.txt" && \
VERSION=$(cat version.txt) && \
wget --no-verbose "https://download3.rstudio.org/ubuntu-14.04/x86_64/shiny-server-$VERSION-amd64.deb" -O ss-latest.deb && \
gdebi -n ss-latest.deb && \
rm -f version.txt ss-latest.deb && \
. /etc/environment && \
R -e "install.packages(c('shiny', 'rmarkdown'), repos='$MRAN')" && \
cp -R /usr/local/lib/R/site-library/shiny/examples/* /srv/shiny-server/
RUN \
apt-get update && apt-get install -y && \
DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="
docker pull shrysr/shiny