Public | Automated Build

Last pushed: 2 years ago
Short Description
Short description is empty for this repo.
Full Description

[[https://travis-ci.org/nilcons/ceh][https://travis-ci.org/nilcons/ceh.png]]

  • Power-user environment to share
  • Agreements and a repository for a shared GNU/Linux power-user environment.
  • So that it's easy to work on shared projects (e.g. the haskell
    header can be =#!/opt/ceh/bin/runhaskell -i /opt/ceh/p/ourproject=,
    instead of some complicated perl one-liner).
  • So that the packages are available with the same version on our
    machines (e.g. =emacs=, =tmux=, =firefox=, etc).
  • These agreements are intentionally based on simple unifications
    instead of abstractions (e.g. =/opt/ceh/home= has to be a symlink
    for the user's home directory, no need to =$HOME= or tilde lookups
    anymore).
  • We use amd64 Nix for package management and if you can decide on the
    distribution that you use as a base system, we recommend amd64 Debian.
    But the point is that this should work on any modern amd64
    GNU/Linux, we also tested Ubuntu trusty and precise.
  • Requirements
  • Modern GNU/Linux distro (ubuntu, debian, etc.).
  • Intel architecture (amd64, no i386/i686 or arm).
  • The machine only has one user who wants to use ceh and nix and she
    will be the owner of the files in =/opt/ceh= and =/nix=.
  • If the machine is running NixOS already, then =/nix= should exist
    already of course and Ceh is able to handle that.
  • Root privileges for the installation. Actually only needed for the
    creation of =/opt/ceh= and =/nix=, not needed for everyday usage or
    package installation.
  • =/etc/passwd= contains your user, so nix doesn't have to use any
    fancy NSS modules.
  • The user uses bash or zsh. This is because he has to source
    =/opt/ceh/scripts/ceh-profile.sh= which supports these two shells.
    Patches are welcome for other shells.
  • Installation instructions
    : sudo mkdir /opt/ceh /nix
    : sudo chown $USER. /opt/ceh /nix
    : chmod 0700 /opt/ceh /nix
    : cd /opt/ceh
    : git clone git://github.com/nilcons/ceh.git .
    : ln -s $HOME home
    : /opt/ceh/scripts/ceh-init.sh
  • Installation instructions for NixOS
    : sudo mkdir /opt/ceh /nix/var/nix/gcroots/auto/ceh
    : sudo chown $USER. /opt/ceh /nix/var/nix/gcroots/auto/ceh
    : chmod 0700 /opt/ceh /nix/var/nix/gcroots/auto/ceh
    : cd /opt/ceh
    : git clone git://github.com/nilcons/ceh.git .
    : ln -s $HOME home
    : /opt/ceh/scripts/ceh-init.sh
  • Everyday usage
    In the =/opt/ceh/bin= directory there are small wrapper scripts. Those
    install the required pacakages for the wrapped binaries on first run.

=/opt/ceh/bin= is put into =$PATH= by =/opt/ceh/scripts/ceh-profile.sh=,
that you're supposed to source from your =~/.bashrc=.

* Exclusions: ignoring programs provided by ceh
If for any reason you prefer to ignore one of our provided binaries,
you can use the =ceh_exclude= script to create a symlink from
=/opt/ceh/bin-user/= to the other version of the binary in your =PATH=.

If you don't want the exclusion anymore, just delete the symlink.

* Removing packages that got installed on-demand
As previously mentioned, the scripts in =/opt/ceh/bin= just install
the programs on demand as you run them. Therefore once ran, they take
up disk space on your machine. If you tried out something big and
would like to get rid of it, then go to =/opt/ceh/installed= and feel
free to delete any symlinks there. After deleting the symlinks, run
=nix-garbage-collect= to free up the disk space. If you decide to use
some deleted software again, it will be redownloaded.

If you delete from =/opt/ceh/installed/essential=, then you will have
to run =ceh-init.sh=, but Ceh will show a warning message on stderr
about that.

* Troubleshooting if something is broken after git pull
Sometimes there are changes that are not backward compatible with the
previous =ceh-init.sh=. In those cases we do our best to provide
warning messages on stderr from scripts, and those messages will ask
you to rerun =ceh-init.sh=. That operation is idempotent, feel free
to run it you think it can fix up something for you.

You can also read the "Upgrade notes" section to know about changes.

* Disable the binary cache, for fun and profit
You may just want to see if it's still possible to build a software
from source, instead of using the nix binary cache. Or the nix binary
cache is down and you still want to install packages instead of having
to wait until it's back.

In both cases, you can just set =$CEH_NO_BIN_CACHE= to 1 and then the
wrapper scripts will start building packages instead of using the
binary cache.

When you do this, you may also want to set =$CEH_BUILD_MAX_JOBS= to
the number of your processor cores in the system, because usually
building of tools (like compilers, linkers, make, etc.) can be
parallelized by nix.

  • Upgrade notes
    * 2017-07-14: no Haskell anymore
    We think that for Haskell, stack is a clear winner in the ecosystem.

Therefore we may add stack support to ceh in the future, but we have
removed our nix based haskell suppotr.

* 2016-04-14: support for multiple outputs
As preparation for a baseline upgrade, we had to do a refactoring: in
new nixpkgs versions, most of the derivations result in multiple
output, not just one. Previously this happened only rarely, so we
handled the special cases with the outFilter hack. This hack is gone,
and instead we refactored the gclink symlinks a little bit to support
multiple derivations.

For the end user, the change shouldn't be notable and everything
should continue working as expected.

* 2014-12-20: no more nixops
NixOps was not working after the 32-bit -> 64-bit migration, I do not
think this is anything to do with the architecture, something has
changed in the bit repo.

Anyways, we know of no users who uses Ceh and nixops together, so it
was easier to remove it. Tell us if you need it!

* 2014-12-20: 32-bit -> 64-bit migration
I love i686, programs are so fluffy and cute. But unfortunately the
world started to ignore that platform and stuff is supported only on
amd64 nowadays. E.g. the haskell "community" ignores i686 altogether
and unit tests routinely break. The nix binary cache is also only
maintained for amd64 and if the glibc is missing for i686, nobody will
fix it and you will have to build everything from source.

So we decided to move ceh to 64-bit, and live with the memory and disk
usage of amd64.

This transition should be seamless from the user point of view, but if
something goes bad, of course you can always just purge Ceh and
reinstall it...

Also, if you want to install 32-bit stuff from some packages, you have
to use the bit32 flag, instead of the old bit64 flag that we had for
64-bit. For Haskell, you can use the envvar CEH_GHC32 to get a 32-bit
GHC and tools.

As a last resort, the base perl and nix package is still i686, so you
can run =ceh-init.sh= even on a i686 system and write your own
wrappers by using the bit32 flag for every package. But we are ready
to remove this convenience the first time it causes some problems.

Sorry guys, i686 is dead.

* 2014-05-08: refactoring
From now on, we don't use profiles for Ceh's internal purposes.
Instead we use the =/opt/ceh/installed= directory.

Profiles essentially give you two features. First, it's a way to
track different states of what is installed and an easy way to roll
back and forward between those states. As we have git, we don't need
this. The other feature is that profiles give you a huge symlink farm
that mimics a directory tree where only the currently installed
packages are seen. We don't need this feature either, because we have
the wrappers in =/opt/ceh/bin= and we can just introduce a constant
symlink farm for the manpages in =/opt/ceh/man=.

This refactoring will also solve the constant issue around file
collissions in profiles and package renames in upstream nixpkgs.

You can safely delete =/nix/var/nix/profiles/ceh= and run a
=nix-garbage-collect= to get rid of some old stuff that never would be
deleted otherwise.

* 2014-03-04: refactoring
The new place for the nixpkgs checkout of Ceh is =/opt/ceh/nixpkgs=,
you can run =rm -rf /nix/var/ceh_nixpkgs= to remove the old cruft.

Also, from now on it's not necessary (or allowed) to specify
derivation hashes when calling =ceh_nixpkgsinstall*= functions.
Derivation hashes are always figured out automatically when on-demand
installations is needed and only the output hash is consulted for
binary consistency.

Another change is that we made ceh-init.sh idempotent and it should
fix up stuff from any state that is reasonable after a git pull or
after deletion of all the profiles in =/nix/var/nix/profiles/ceh=.
This means that =ceh-upgrade.sh= is no more and you should always use
=ceh-init.sh= if you think Ceh should be reinitialized.

* 2014-01-07: nix channels moved
On existing installations, run the following:
: nix-channel --remove nixpkgs
: nix-channel --add http://nixos.org/channels/nixpkgs-unstable
: nix-channel --update

* 2014-02-03: xpra upgrade broken
Unfortunately the nixpkgs guys changed the name of the xpra packages
from python2.7-xpra to xpra. As described in the Troubleshooting
section, this is causes a filename collision. This can be fixed:
: /opt/ceh/bin/nix-env -p /nix/var/nix/profiles/ceh/bin -e python2.7-xpra
: /opt/ceh/bin/xpra

* 2013-08-05: ~/.nix-profile reserved for the user
The =/nix/var/nix/profiles/per-user/root/profile= profile (which is
symlinked by default from =$HOME/.nix-profile=) is not used for
installations anymore. Earlier versions of ceh used to install
packages there, but now that profile is reserved for the user.

This is good, because if you need some package (e.g. =vim=) quickly
from the nixpkgs version that is current the ceh baseline, you can
just do:
: ceh_nixpkgs_install vim
: ~/.nix-profile/bin/vim

Alternatively, you can use the most up-to-date nixpkgs:
: nix-channel --add http://nixos.org/channels/nixpkgs-unstable
: nix-channel --update
: nix-env -i vim
: ~/.nix-profile/bin/vim

No need to create a wrapper in =/opt/ceh= just for a quick test. If
you want to get rid of your user profile, simply delete
=/nix/var/nix/profiles/per-user/$USER/profile=. This is from now on
is totally separate from ceh's own nix profiles. Those are located in
=/nix/var/nix/profiles/ceh/
=.

If you're upgrading =ceh= from a version, where the default profile
was used for package installations, please delete
=/nix/var/nix/profiles/per-user/root/profile*=.

* 2013-08-15: renames
nix-init.sh -> ceh-init.sh
nix-profile.sh -> ceh-profile.sh
nix-purge.sh -> ceh-purge.sh
nix-upgrade.sh -> ceh-upgrade.sh

And install-world.sh and predict-binary-cache.sh moved to scripts/maintainer.

  • TODOs
  • Have an LD_PRELOAD library that hides =/usr/lib/mozilla/plugins= from Firefox's view.
  • Design and other info for Ceh hackers, maintainers
    * Directory structure
    =/opt/ceh/bin/=: wrapper scripts, they make sure that the wanted
    package is already installed into =/nix/store= and symlinked from
    =/opt/ceh/installed=. After this initialization they just exec
    the binary from =/nix/store=,

=/opt/ceh/bin-user/=: gitignored directory for local exclusions, so the
end-user has the final say. The =scripts/ceh-profile.sh= script puts
this directory in front of the =bin/= directory in PATH,

=/opt/ceh/emacs.d/=: contains =nix-mode.el= only, we used to
distribute emacs modules via Ceh, but it didn't work too well, it's
simpler to just use melpa,

=/opt/ceh/home=: gitignored symlink to the user's home, this is
extremely useful, because if Ceh is installed, you can simply use
=/opt/ceh/home= as a path to point to the user's home directory
without using the tilde expansion of the shell (that doesn't work in
syscalls or on the shebang line),

=/opt/ceh/lib/=: the Ceh (mainly written in Perl) internals,

=/opt/ceh/p/=: gitignored symlinks to the user's projects, so every project can
have an absolute symlink in =/opt/ceh/p/whatever-project=, like we
have =/opt/ceh/home= as an absolute path for the user's home directory,

=/opt/ceh/scripts/=: Ceh end-user scripts (install, purge, profile, upgrade),

=/opt/ceh/scripts/maintainer=: Ceh maintainer scripts,

=/opt/ceh/nixpkgs=: contains checked out versions of the nixpkgs
package collection, used and maintained by =lib/CehInstall.pm=,
gitignored,

=/nix=: not in Git, but doesn't store any important data. Deleting
all the content and reinstalling Ceh will result in some compilation
and downloading, but everything will be recreated eventually,

=/nix/store=: contains the installed package roots. Managed by nix
commands, like =nix-store=, =nix-env= and =nix-instantiate=,

=/opt/ceh/installed/packages=: gitignored directory storing the currently
installed packages on a machine. The state is stored as symlinks to
=/nix/store= and there are symlinks to here from
=/nix/var/nix/gcroots/auto/ceh=. So the purpose of this directory is
two-fold: first, it protects the packages that the user use from being
garbage collected. Secondly, we can determine with a simple =stat=
whether a package is already installed.

=/opt/ceh/installed/essential=: nix and perl that is used by Ceh,

=/opt/ceh/installed/tools=: the tools used by Ceh itself (e.g. =which=).

* Adding a new package to ceh

  • Simply create a wrapper script in =/opt/ceh/bin=, use =tmux= as an example,
  • use the autoinit feature to get the initial hash values:
    : ceh_nixpkgs_install_bin('newpkg', AUTOINIT);
    After running this wrapper once, the AUTOINIT will be replaced with
    the current nixpkgs version and out hash value.
  • add it to =scripts/maintainer/install-world.sh=,
  • symlink manpages in =/opt/ceh/man= if appropriate,
  • send a github pull request.

* Upgrading ceh to a new baseline

  • Update CEH_BASELINE_NIXPKGS in =lib/CehBase.pm= with a new hash,
    that you should get from http://hydra.nixos.org/jobset/nixpkgs/trunk, by
    choosing the most recent build that has no pending (gray) packages anymore.
    Clicking inputs, you can get the git revision you need.

  • Update the used Nix and Perl to the new baseline:
    : CEH_AUTO_UPGRADE=1 \
    : ENSURE_BASE_PERL=/opt/ceh/installed/essential/perl.32/MAIN/bin/perl \
    : ENSURE_BASE_NIXPATH=/opt/ceh/installed/essential/nix.32/MAIN /opt/ceh/lib/ensure_base_installed.pl

  • Update CEH_BASELINE_NIXPATH in =lib/CehBase.pm= (peek from CehInstall.pl:ensure_base_installed).

  • Update CEH_BASELINE_PERL in =lib/CehBase.pm= (peek from CehInstall.pl:ensure_base_installed).

  • Update =emacs.d/nix-mode.el=.

  • Update all the wrappers to have the new baseline versions. For
    example to upgrade git:
    : export CEH_AUTO_UPGRADE=1
    : /opt/ceh/bin/gitk

    This will update the hash in lib/Packages/Git.pm.

  • To update all the packages, use scripts/maintainer/install-world.sh.
    You may also find scripts/maintainer/predict-binary-cache.sh useful
    in selecting and checking a baseline.

  • =git grep= the old baseline hash to see that everything has been
    updated.

  • git commit, git push.

* Upgrading nix inside ceh-init.sh
This should only be done if you have some strong reason for this.
E.g. it's no longer compatible with current glibc on standard linuxes
or the communication protocol for nix-daemon changed and therefore Ceh
is unusable on NixOS.

  • Choose a new release from http://nixos.org/nix/download.html, read
    release notes, etc.

  • Look into the install shell script, to figure out the current URL
    for i686-linux, e.g. for 1.11.2 it was
    https://nixos.org/releases/nix/nix-1.11.2/nix-1.11.2-i686-linux.tar.bz2

  • tar xvfj nix-x.x.x-i686-linux.tar.bz2

  • Update =CEH_NIX_DOWNLOAD=, =NIX_TARDIR_NAME=, =CEH_NIX= and
    =CEH_NIXPERL= in =scripts/ceh-init.sh=.

  • Test if =scripts/ceh-init.sh= still works on a clean Debian and on a
    clean NixOS.

  • git commit, git push.

  • Techtalk
    #+BEGIN_SRC
    Title: Nix & ceh: reproducible power-user environment

Abstract:

It's common for engineers to use multiple GNU/Linux based computers
daily, however it's not easy to achieve the same working environment
on each of them.

Computers owned by the company may run the company's linux variant,
while computers at home run some common distribution. Some of the
running variants are already 64-bit, some are not. Some of them is 2
years old, some of them is fresh.

On top of this mixture, it's hard to imagine to run exactly the same
version of GNU Emacs or Mozilla Firefox (with all the hard to install
plugins: flash, googletalk, java), without installing a common base
operating system.

This is the problem that I wanted a solution for, when I looked into
the Nix package manager and the Nixpkgs package collection. After a
month of exploration I now have a working setup (called ceh), that I
run on all of my computers to synchronize (via Git) the choice of
software between them.

In the techtalk I will present the fundamentals of Nix and show ceh,
my setup around Nix.

#+END_SRC

Docker Pull Command
Owner
nilcons
Source Repository