Automate builds in DockerHub for KaveToolbox, builds at every push to master.
317
Data analytics toolkit part of the KAVE, installed with AmbariKave, and also installable stand-alone http://beta.kave.io, a wiki for the entire KAVE is maintained on the cluster installer, AmbariKave wiki
Examples:
Why is this complicated?
What errors might I see?
Fix:
Example:
sudo su
source /opt/KaveToolbox/pro/scripts/KaveEnv.sh
conda update conda
pip install pymongo
Our libraries:
Installer for:
Examples of:
CentOS6, CentOS7, Redhat7, Ubuntu 14 and Ubuntu 16 are used for testing, although no guarantees are given.
Only bash as a default shell is supported at the moment, users with a different default have reported many problems.
Please get in touch if you would like to make enquiries about this.
KaveToolbox is aimed at making the installation of our key analytics software and libraries seamless so that one-click deployment is possible and encouraged, taking the pain out of working out prerequisites, compilation, for most of our software. When you just want to get stuck straight into the data, you can bring along your same toolbox. It ensures a common environment to allow for simpler code distribution across all data nodes "fire and forget" instead of "push and pray".
KaveToolbox recognises two types of distribution:
Node: 5 GB of disk space for the software, additional 2 GB temp space needed during installation
Workstation: 7 GB of disk space for the software, additional 2 GB temp space needed during installation
Node: 1 core 2GB of RAM
Workstation: 2 core 4GB RAM
An internet connection (many packages will be downloaded form various sites)
Centos6/7 review your yum.conf file to make sure you are not ignoring certain packages from being installed
Nodes are likely to have even higher requirements for other service requirements such as Hadoop or storm.
We also release the software packaged within docker containers. See http://hub.docker.com/r/kave/kavetoolbox. For example:
docker run -it kave/kavetoolbox:3.0-Beta.c7.node /bin/bash
When making a local installation you have two choices:
We recommend to install with the default configurations, but in case you want to modify the configurations you can create a file in /etc/kave/CustomInstall.py,For an example and more information run the installer with --help
yum -y install wget curl tar zip unzip gzip python
wget http://repos:[email protected]/noarch/KaveToolbox/3.0-Beta/kavetoolbox-installer-3.0-Beta.sh
sudo bash kavetoolbox-installer-3.0-Beta.sh [--quiet]
(--quiet is for a quieter install, remove the brackets!) Remember the help at this stage [--help] ( NB: yum is the standard package manager for Centos/redhat. To install on Ubuntu the equivalent is apt-get )
( NB: the repository server uses a semi-private password only as a means of avoiding robots and reducing DOS attacks this password is intended to be widely known and is used here as an extension of the URL )
#test ssh keys with
ssh -T [email protected]
#if this works,
git clone [email protected]:KaveIO/KaveToolbox.git
#then install with
sudo ./KaveToolbox/scripts/KaveInstall [--quiet]
(--quiet is for a quieter install, remove the brackets!) Remember the help at this stage [--help]
cd /opt/KaveToolbox/pro/examples
ipython notebook
And/or visit http://nbviewer.ipython.org/
Optional: Editing configuration files
Optional: Set mirrors/nearside cache
Optional: Additional installation options
Troubleshooting:
There are three possible update mechanisms
sudo /opt/KaveToolbox/pro/scripts/KaveUpdate --list
sudo /opt/KaveToolbox/pro/scripts/KaveUpdate --help
sudo /opt/KaveToolbox/pro/scripts/KaveUpdate --quiet
The update script works well for updating between 2.X versions, and can also be used for 1.X, but only with either:
The --clean-after flag is a common addition to the update to remove deprecated software after install
If you are trying to upgrade from 1.X to 2.X, either use --clean-before to remove the previous install, or move /opt/KaveToolbox/ to /opt/KaveToolbox/1.X and /opt/anaconda to /opt/anaconda/2.2 before installation
The correct paths to directly use our tools will be automatically added to your environment provided:
In other cases you will need to get/set environment manually
source [directory, e.g. /opt/KaveToolbox/pro/scripts]/KaveEnv.sh
the ASCII-art KAVE banner only shows up for interactive, non-dumb terminals, to turn off the KAVE banner even in that case do
touch ~/.nokaveBanner
To disable automatic setting of the environment for this user:
touch ~/.nokaveEnv
To force setting the environment for this user in case they would normally be skipped, first remove .nokaveEnv, then:
touch ~/.kaveEnv
cd $KAVETOOLBOX/examples
ipython notebook
--> Choose, for example, rootnotes.ipynb
--> Kernel --> Restart
--> Cells --> RunAll
Migration to python3 as default
Ideally all of your nodes will have access to the internet during installation in order to download software.
If this is not the case, you can, possibly, implement a near-side cache/mirror of all required software. This is not very easy, but once it is done one time, you can keep it for later.
To setup a local near-side cache for the KAVE tool stack is quite easy. First either copy the entire repository website to your own internal apache server, or copy the contents of the directories to your own shared directory visible from every node.
mkdir -p /my/shared/dir
cd /my/shared/dir
wget -R http://repos:[email protected]/
Then create a /etc/kave/mirror file on each node with the new top-level directory to try first before looking for our website:
echo "/my/shared/dir" >> /etc/kave/mirror
echo "http://my/local/apache/mirror" >> /etc/kave/mirror
So long as the directory structure of the nearside cache is identical to our website, you can drop, remove or replace, any local packages you will never install from this directory structure, and update it as our repo server updates.
You might consider creating a near-side cache, and/or configuring your proxy settings correctly, since we use wget for the downloads, your existing proxy settings (e.g. HTTP_PROXY environment variable) should be sufficient.
Don't forget that the root user/sudo also must comminicate over the proxy, and this may mean propagating the right environment variables. Try adding:
Defaults env_keep +="http_proxy"
Defaults env_keep +="https_proxy"
to your sudoers file with visudo
from here
We can't trouble shoot your networking issues for you, but if you are trying to install from behind a proxy, check the "How can I install behind a proxy" FAQ, also talk with your network administrator and decide if you need to setup a nearside cache.
cd /root/install/location
./configure [options e.g. linuxx8664gcc --enable-python --enable-mathmore --enable-minuit2 --enable-roofit --fail-on-missing]
make -j numcores
or follow the instructions on the root website to install root yourself
Many different packages are needed, did you maybe run out of space? Or did you ignore kernel packages in your yum.conf?
Check /etc/yum.conf and see if there is anything your are ignoring or forbidding from installing.
This is gnome trying to spawn an x window to have you enter your password. Work around by:
unset SSH_ASKPASS
So long as the pre-requisites are already installed (see the yum install commands in the kaveconfiguration.py) it is possible to install all the software we package into a local directory, however that is not implemented yet and will not permit seamless integration of all users and machines in a network, and it will not be possible to automatically source the environment for all users.
This is usually your local browser which is blocking things:
In the first case, you can simply permit scripts running, by clicking on the correct icon and choosing the correct option.
In the second case, there are tow solutions. Best is to restart/modify mpld3 options with the correct options to switch to https for the javascript part aswell. The other option is to allow mixed content in your browser for ipython notebooks.
http://stackoverflow.com/questions/21089935/unable-plot-with-vincent-in-ipython https://mpld3.github.io/modules/API.html https://mpld3.github.io/modules/API.html#mpld3.enable_notebook
On Centos7, for some reason the vnc installation/start does not be default recognise the gnome installation
To fix this, edit your .vnc/xstartup file to contain:
#!/bin/sh
[ -r /etc/sysconfig/i18n ] && . /etc/sysconfig/i18n
export LANG
export SYSFONT
vncconfig -iconic &
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
OS=`uname -s`
if [ $OS = 'Linux' ]; then
case "$WINDOWMANAGER" in
*gnome*)
if [ -e /etc/SuSE-release ]; then
PATH=$PATH:/opt/gnome/bin
export PATH
fi
;;
esac
fi
if [ -x /etc/X11/xinit/xinitrc ]; then
exec /etc/X11/xinit/xinitrc
fi
if [ -f /etc/X11/xinit/xinitrc ]; then
exec sh /etc/X11/xinit/xinitrc
fi
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
twm &
This is caused by an environment variable being inherited form one user to the next. Simple fix, unset $XDG_RUNTIME_DIR .
In some cases we have seen that users have a file ~/.local/share/jupyter/kernels/python2/kernel.json where the wrong python executable is given.
Easy fix, change the name of the python executable to simply 'python' in this file.
Content type
Image
Digest
Size
2.9 GB
Last updated
about 10 years ago
docker pull kave/kavetoolbox-head:latest.u14.workstation