shrysr/shiny

By shrysr

Updated about 5 years ago

Image
0

307

Readme on Readthedocs:

Documentation Status

Docker image Cloud Build Status. Note: Sometimes images are built locally and pushed dockerhub

Docker Asmith Cloud Build StatusDocker Cloud Build StatusDocker Shiny Cloud Build StatusDocker Cloud Build Status

Build Status

Build Status

TL;DR

  • The Init section will grow to contain everything that you need to know this project and get started with using the tools.
  • The easiest way at the moment to test-drive these containers is via the Matrix DS platform. Here is a project you can forklift, that has the shiny image added as a custom tool that can be launched.
  • The notes section contains additional notes gathered while developing this project.
  • Archive contains test and initial configurations that is essentially a sandbox for future projects. These may not be exported in the Readme.md. However these will be available in the Readme.org file
  • One alternate method currently available to read the documentation is via readthedocs

Quick Commands:

Docker Images:

docker pull shrysr/shiny:v2
docker pull shrysr/rstudio:v2

Docker compose to launch services:

docker compose up -d rstudio
docker compose up -d shiny
# To launch both : docker compose up -d

Init

The starting point of this project was Matt Dancho's shinyauth docker file, which then expanded into adapting the well designed Matrix DS tools for my purpose. Their stack of docker based tools is replicated here with my customization which adds libraries and other functionality.

The goal is to develop a workflow based on Docker (and other tools) to create a reproducible, standard, consistent environment to run a variety of datascience projects, with different development and production environments. In particular, I want to be able to develop and deploy dashboards like shiny or streamlit.io quickly and with ease.

This Readme consists of the entire documentation and source code, maintained in a single Org mode document which is used to regenerate and manage the entire source code and the exports to markdown and rst. i.e Literate Programming in all it's splendor and warts.

Why not just use the MatrixDS stack directly and add the missing packages as layers?

In a sense, that is what I did, except that I constructed my own base image instead of relying on modifying a MatrixDS image. I also wanted to build these images by hand as my set of tools, even if the tools were largely similar to the MatrixDS stack. From whatever I've learned of Docker - the MatrixDS stack is quite efficient and the cascading + common dependency layer makes sense to use. There may be other methods, but this certainly appeared technically sensible.

The main containers to be aware of, and also hosted on dockerhub are :

  1. shrysr/asmith:v1
  2. shrysr/rbase:v2
  3. shrysr/rstudio:v2
  4. shrysr/shiny:v2
  5. One of the earlier versions created is at shrysr/datasciencer, however, this has been superceded by the above images, and may grow into something different.

Note the tags to be used with the images.

The rbase image is built on the first asmith image. The RStudio and Shiny images are based of a common rbase dependency environment. However, additional packages can be specified for these, and it is not necessary to rebuild the rbase layer each time.

img

Launching the docker containers

The easiest way to launch is to use the docker-compose file.

Starting an rstudio service is as simple as:

docker-compose up rstudio

Replace rstudio with shiny above to launch the shiny server.

Note that the docker-compose setup involves using a fully specified project path. This is set to a generic path and the system has been tested on a Mac OS and several compute instances of Linux running Debian and Ubuntu.

Docker Compose setup

The purpose of the docker container to is make the environment variables and other settings like volume names and ports to be conveniently configured.

There are 2 services : rstudio server and a shiny server. The default up command will launch both of them. Individual services can be launched using their names.

It would b a good practice to name the containers based on the project so as to be able to distinguish different containers.

version: "2"

services:
  rstudio:
    image: shrysr/rstudio:v2
    container_name: rstudio_s
    environment:
      - PASSWORD=shreyas
      - DISABLE_AUTH=false
      - WORKDIR="/home/rstudio/"
    restart: always
    volumes:
      - /Users/shrysr/docker-testing-sr/:/home/rstudio/
    ports:
      - "8787:8787"

  shiny:
    image: shrysr/shiny:v2
    container_name: shiny
    ports:
      - "3838:3838"
    restart: always
    volumes:
     - /Users/shrysr/docker-testing-sr/shiny-server/:/srv/shiny-server/

YAML

Travis

This travis file uses the docker service specification to start the docker service. This can be viewed by the systemctl start command in the logs. The build is designed in stages, due to the cascading image dependency. However, the rstudio and shiny images can be constructed in parallel and actually do not take long.

Note that the entire build does not complete on Travis at the moment because of limitations in the free tier of computing time. However, the builds do complete on Github Acions described subsequently.

services:
  - docker

jobs:
  include:
    - stage: asmith
      script: docker build asmith/. -t shrysr/asmith:v1
    - stage: rbase
      script: docker build rbase/. -t shrysr/rbase:v2
    - stage: rstudio-and-shiny
      script: docker build rstudio/. -t shrysr/rstudio:v2
      script: docker build shiny/. -t shrysr/shiny:v2

Github Workflows

ASmith YAML for CI with github
name: Docker Image CI

on:
  push:
    paths:
    - '!/docs/*'
    - '!Readme.*'
    - '!*.md'
    - '!*.org'

jobs:

  build:

    runs-on: ubuntu-18.04

    steps:
    - uses: actions/checkout@v1
    - name: Build Asmith
      run: docker build asmith/. --file asmith/Dockerfile --tag my-image-name:$(date +%s)
rbase YAML for CI with github
name: Docker Image CI

on:
  push:
    paths:
    - '!/docs/*'
    - '!Readme.*'
    - '!*.md'
    - '!*.org'

jobs:

  build:

    runs-on: ubuntu-18.04

    steps:
    - uses: actions/checkout@v1
    - name: Build rbase
      run: docker build rbase/. --file rbase/Dockerfile --tag my-image-name:$(date +%s)
Rstudio YAML for CI with github
name: Docker Image CI

on:
  push:
    paths:
    - '!/docs/*'
    - '!Readme.*'
    - '!*.md'
    - '!*.org'

jobs:

  build:

    runs-on: ubuntu-18.04

    steps:
    - uses: actions/checkout@v1
    - name: Build rstudio
      run: docker build rstudio/. --file rstudio/Dockerfile --tag my-image-name:$(date +%s)
Shiny YAML for CI with github
name: Docker Image CI

on:
  push:
    paths:
    - '!/docs/*'
    - '!Readme.*'
    - '!*.md'
    - '!*.org'

jobs:

  build:

    runs-on: ubuntu-18.04

    steps:
    - uses: actions/checkout@v1
    - name: Build shiny
      run: docker build shiny/. --file shiny/Dockerfile --tag my-image-name:$(date +%s)

ASmith

This is the very first layer. This layer adds several OS packages and starts with a specific version of Ubuntu (v18.04). Currently, it is largely left the same except for adding the package dtrx, which is useful to quickly zip and unzip files.

This layer does not take very long to build, however, if it is - then all the other subsequent layers will probably need to be rebuilt.

Dockerfile

FROM ubuntu:18.04

LABEL maintainer="Shreyas Ragavan <sr@eml.cc>" \
	version="1.0"

USER root

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update

# Install all basic OS dependencies
RUN apt-get update \
  && apt-get install -yq --no-install-recommends \
    apt \
    apt-utils \
    bash-completion \
    build-essential \
    byacc \
    bzip2 \
    ca-certificates \
    emacs \
    file \
    flex \
    fonts-dejavu \
    fonts-liberation \
    fonts-texgyre \
    g++ \
    gcc \
    gettext \
    gfortran \
    git \
    gnupg2 \
    gsfonts \
    hdf5-tools \
    icu-devtools \
    jed \
    lmodern \
    locales \
    make \
    mesa-common-dev \
    nano \
    netcat \
    openjdk-8-jdk \
    pandoc \
    software-properties-common \
    sudo \
    texlive-fonts-extra \
    texlive-fonts-recommended \
    texlive-generic-recommended \
    texlive-latex-base \
    texlive-latex-extra \
    texlive-xetex \
    tzdata \
    unzip \
    vim \
    wget \
    zip \
	libsodium-dev \
  && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
  && locale-gen en_US.utf8 \
  && /usr/sbin/update-locale LANG=en_US.UTF-8

# make the "en_US.UTF-8" locale so postgres will be utf-8 enabled by default
ENV LANG=en_US.utf8 \
    LC_ALL=en_US.UTF-8 \
    TERM=xterm \
    APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1

# Install additional libraries
RUN apt-get install -yq --no-install-recommends \
    libblas-dev \
    libcurl4 \
    libcurl4-gnutls-dev \
    libgdal-dev \
    libglu1-mesa-dev \
    libgmp3-dev \
    libicu60 \
    libjpeg-turbo8 \
    libmagick++-dev \
    libmariadb-client-lgpl-dev \
    libmpfr-dev \
    libmpfr-dev \
    libncurses5-dev \
    libnettle6 \
    libnlopt-dev \
    libopenblas-dev \
    libpango1.0-0 \
    libpangocairo-1.0-0 \
    libpng16-16 \
    libpq-dev \
    libsasl2-dev \
    libsm6 \
    libssl-dev \
    libtiff5 \
    libtool \
    libudunits2-dev \
    libxext-dev \
    libxml2-dev \
    libxrender1 \
    zlib1g-dev \
	dtrx

# Set timezone noninteractively
RUN ln -fs /usr/share/zoneinfo/US/Pacific /etc/localtime

# Python stuff
RUN apt-get install -y --no-install-recommends \
    python-pip \
    python-setuptools \
    python-wheel \
    python-dev \
    python3-pip \
    python3-setuptools \
    python3-wheel \
    python3-dev \
  && apt-get clean

#install git, vim

RUN apt-get install -y git \
	                   vim \
                       curl

#install kaggle cli
RUN pip install kaggle dvc tensorflow keras pandas

#mongo cli
RUN apt-get install -y mongodb-clients

#mysql shell
RUN apt-get install -y mysql-client

#postgre shell
RUN apt-get install -y postgresql-client

# Add Tini
ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

RUN apt-get clean \
  && rm -rf /var/lib/apt/lists/*

rbase

This layer contains all the basic R packages required for datascience and ML. A bunch of packages were added to the already extensive default list of packages in MatrixDS's docker file.

The packages are defined in an R script called packages.R.

This layer takes a tremendously long time to build. A couple of hours on a Macbook Pro 2019, with 6 cores and 32 GB of RAM. One should be careful in assessing whether this layer has to be disturbed. Automated builds on Dockerhub are likely to take even longer.

Note: As such the dockerfile indicates that the packages are called in the last 2 layers only. It may be possible that subsequent image builds do not take as much time as I imagine.

  • It may be easier to find a way to keep the additional packages specified in the rstudio and shiny package list to be in sync.

R package list - BASE

This is a list of the basic packages being installed. These conver many commonly used libraries for data science. This layer will take a Long time to install.

Do not install custom libraries to this layer. Install in the next layer.

#Script for common package installation on MatrixDS docker image
p<-c('nnet','kknn','randomForest','xgboost','tidyverse','plotly','shiny','shinydashboard',
	  'devtools','FinCal','googleVis','DT', 'kernlab','earth',
     'htmlwidgets','rmarkdown','lubridate','leaflet','sparklyr','magrittr','openxlsx',
     'packrat','roxygen2','knitr','readr','readxl','stringr','broom','feather',
     'forcats','testthat','plumber','RCurl','rvest','mailR','nlme','foreign','lattice',
     'expm','Matrix','flexdashboard','caret','mlbench','plotROC','RJDBC','rgdal',
     'highcharter','tidyquant','timetk','quantmod','PerformanceAnalytics','scales',
     'tidymodels','C50', 'parsnip','rmetalog','reticulate','umap', 'glmnet', 'easypackages', 'drake', 'shinythemes', 'shinyjs', 'recipes', 'rsample', 'rpart.plot', 'remotes', 'DataExplorer', 'inspectdf', 'janitor', 'mongolite', 'jsonlite', 'config' )


install.packages(p,dependencies = TRUE)

R Package list - CUSTOM

Add your custom packages to this layer. In this way, only the additional packages are installed in a new layer.

#Script for common package installation on MatrixDS docker image
PKGS <- c(
      "tidyverse", "mapproj", "maps", "genius", "shinycssloaders", "gmailr"
)

install.packages(PKGS, dependencies = TRUE)

# These packages are sometimes not available for the current R version
# , and therefore installed directly from github
devtools::install_github("tidyverse/googlesheets4", dependencies = TRUE)
devtools::install_github("PMassicotte/gtrendsR", dependencies = TRUE)

Dockerfile

FROM shrysr/asmith:v1

LABEL maintainer="Shreyas Ragavan <sr@eml.cc>" \
	version="1.0"

#install some helper python packages
RUN pip install sympy numpy

# R Repo, see https://cran.r-project.org/bin/linux/ubuntu/README.html
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/' >> /etc/apt/sources.list
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
RUN add-apt-repository ppa:marutter/c2d4u3.5

# R-specific packages
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    r-base \
    r-base-core \
    r-recommended \
    r-base-dev \
    r-cran-boot \
    r-cran-class \
    r-cran-cluster \
    r-cran-codetools \
    r-cran-foreign \
    r-cran-kernsmooth \
    r-cran-matrix \
    r-cran-rjava \
    r-cran-rpart \
    r-cran-spatial \
    r-cran-survival

COPY packages.R /usr/local/lib/R/packages.R
COPY custom_packages.R /usr/local/lib/R/custom_packages.R

# Install Basic R packages for datascience and ML
RUN R CMD javareconf && \
    Rscript /usr/local/lib/R/packages.R

# Though this has been installed upstream, apprently it has to be setup again.
RUN apt-get update \
&& apt-get install -y --no-install-recommends libsodium-dev

# Install custom set of R packages. This is on a separate layer for efficient image construction
RUN Rscript /usr/local/lib/R/custom_packages.R

*

Shiny

Overview of the process:

Suppose you have a project folder within which related scripts, shiny apps, etc live. This directory is mounted as a volume to the docker container. The docker container will check for the presence of a folder called shiny-server and if not available, will create it. Even if the folder is available, the contents of test_apps will be copied into the image.

Into the shiny-server folder, the test_apps folder containing shiny apps for testing are copied.

Environment and Profile

R_LIBS=/usr/local/lib/R/site-library:/usr/local/lib/R/library:/usr/lib/R/library:/srv/R/library

.libPaths("/srv/R/library/")

# Things you might want to change
# options(papersize="a4")
# options(editor="notepad")
# options(pager="internal")

# R interactive prompt
# options(prompt="> ")
# options(continue="+ ")

# to prefer Compiled HTML
help options(chmhelp=TRUE)
# to prefer HTML help
# options(htmlhelp=TRUE)

# General options
options(tab.width = 4)
options(width = 130)
options(graphics.record=TRUE)

.First <- function(){
 library(Hmisc)
 library(R2HTML)
 cat("\nWelcome at", date(), "\n")
}

.Last <- function(){
 cat("\nGoodbye at ", date(), "\n")
}

app.r

#
# This is a Shiny web application on MatrixDS.
#
# Find out more about building applications with Shiny here:
#
#    http://shiny.rstudio.com/
#

##########################################################################################
# This points the Shiny server tool to any libraries installed with RStudio
# that means that any library you install on your RStudio instance in this project,
# will be available to the shiny server
##########################################################################################

.libPaths( c( .libPaths(), "/srv/.R/library") )

##########################################################################################
# Here you can call all the required libraries for your code to run
##########################################################################################

library(shiny)

##########################################################################################
# For deploying tools on MatrixDS, we created this production variable
# when set to true, your shiny app will run on the shiny server tool upon clicking open
# when set to false, your shiny app will run when you hit the "Run App" button on RStudio
##########################################################################################

production <- TRUE

##########################################################################################
# The shiny server tool uses a different absolute path than RStudio.
# this if statement denotes the correct path for the 2 values of the production variable
##########################################################################################

if(production == FALSE) {
  #if you using the RStudio tool
  shiny_path <- "~/shiny-server/"
  home_path <- "~/"
} else {
  #if you are using the shiny tool
  shiny_path <- "/srv/shiny-server/"
  home_path <- "/srv/"
}

##########################################################################################
# To call a file/artifact in your MatrixDS project use the following line of code
# this example uses the function read.csv
#  my_csv <- read.csv(paste0(home_path,"file_name.csv"))
##########################################################################################

# Define UI for application that draws a histogram
ui <- fluidPage(

   # Application title
   titlePanel("Old Faithful Geyser Data"),

   # Sidebar with a slider input for number of bins
   sidebarLayout(
      sidebarPanel(
         sliderInput("bins",
                     "Number of bins:",
                     min = 1,
                     max = 50,
                     value = 30)
      ),

      # Show a plot of the generated distribution
      mainPanel(
         plotOutput("distPlot")
      )
   )
)

# Define server logic required to draw a histogram
server <- function(input, output) {

   output$distPlot <- renderPlot({
      # generate bins based on input$bins from ui.R
      x    <- faithful[, 2]
      bins <- seq(min(x), max(x), length.out = input$bins + 1)

      # draw the histogram with the specified number of bins
      hist(x, breaks = bins, col = 'darkgray', border = 'white')
   })
}

# Run the application
shinyApp(ui = ui, server = server)

shiny server script

This is script to execute or run the shiny server. Apparently, it is necessary to be called via script in this fashion for the process to work, rather than the docker file itself. In a way this helps keeping the code modular. It is generally unlikely any changes would be needed here.

#!/bin/sh

# Make sure the directory for individual app logs exists
mkdir -p /var/log/shiny-server
chown shiny.shiny /var/log/shiny-server

if [ "$APPLICATION_LOGS_TO_STDOUT" = "false" ];
then
    exec shiny-server 2>&1
else
    # start shiny server in detached mode
    exec shiny-server 2>&1 &

    # push the "real" application logs to stdout with xtail
    exec xtail /var/log/shiny-server/
fi

packages

#Script for common package installation on MatrixDS docker image
p<-c('reticulate')


install.packages(p,dependencies = TRUE)

version

Dockerfile

The folder test_apps will contain shiny apps meant to test functionality. This is copied into the docker image.

  • [2020-01-08 Wed] During the image build, there were messages that the rmarkdown and shiny libraries could not be installed for this version of R. However, the shiny apps do display in the browser. This needs to be investigated.

Changes: Reduced a step and added the tree package. This makes it easier to troubleshoot.

FROM shrysr/rbase:v2

LABEL maintainer="Shreyas Ragavan <sr@eml.cc>" \
	version="2.0"

COPY packages.R /usr/local/lib/R/packages.R

#install R packages
RUN R CMD javareconf && \
    Rscript /usr/local/lib/R/packages.R

RUN apt-get update && apt-get install -y \
    gdebi-core \
    pandoc \
    pandoc-citeproc \
    libcurl4-gnutls-dev \
    libcairo2-dev \
    libxt-dev \
    xtail \
	tree

COPY entrypoint.sh /entrypoint.sh
RUN mkdir -p /root/shiny-server/  \
	&&  mkdir -p /root/shiny-server/test_shiny/

COPY test_apps/ /root/shiny-server/test_shiny/


# Download and install shiny server
RUN wget --no-verbose https://download3.rstudio.org/ubuntu-14.04/x86_64/VERSION -O "version.txt" && \
    VERSION=$(cat version.txt)  && \
    wget --no-verbose "https://download3.rstudio.org/ubuntu-14.04/x86_64/shiny-server-$VERSION-amd64.deb" -O ss-latest.deb && \
    gdebi -n ss-latest.deb && \
    rm -f version.txt ss-latest.deb && \
    . /etc/environment && \
    R -e "install.packages(c('shiny', 'rmarkdown'), repos='$MRAN')" && \
    cp -R /usr/local/lib/R/site-library/shiny/examples/* /srv/shiny-server/

RUN \
  apt-get update && apt-get install -y && \
  DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="

Docker Pull Command

docker pull shrysr/shiny