bugwz/ceph-dev

By bugwz

•Updated over 3 years ago

Docker image for ceph dev

Image

284

Overview Tags

bugwz/ceph-dev repository overview

⁠Description

Docker image for ceph dev:

Build ceph (pacific , quincy, reef)
Package ceph to RPMs (pacific , quincy, reef)
Deploy ceph by ceph-ansible (stable-6.0 , stable-7.0)
RPM web server by Nginx

⁠Environment

⁠System Env

CentOS 8 ( 8.5.2111 )
Kernel 5.4.0

⁠Python Virtual Env

ceph-pacific : Build and package the pacific version of ceph.
- Python 3.8.16
- Pip 23.0
ceph-quincy : Build and package the quincy version of ceph.
- Python 3.8.16
- Pip 23.0
ceph-reef : Build and package the reef version of ceph.
- Python 3.8.16
- Pip 23.0
ceph-ansible-stable-6.0 : Deploy the pacific version of ceph by the stable-6.0 branch of ceph-ansible.
- Python 3.8.16
- Pip 23.0
- Ansible 2.10.7
ceph-ansible-stable-7.0 : Deploy the quincy version of ceph by the stable-7.0 branch of ceph-ansible.
- Python 3.8.16
- Pip 23.0
- Ansible 2.12.10

⁠Usage

Start docker image:

docker run -it --privileged=true -v "/data/:/data/tmp" -p 80:80 bugwz/ceph-dev:latest /bin/bash

--privileged=true : Allow sysctl to change configs in docker container.
-v /data/:/data/tmp : The mapping directory inside docker must be /data/tmp.
-v 80:80 : When using ceph-ansible to deploy a custom ceph version, we need to use rpm's web service.

⁠Build Ceph

Set git_repo and git_branch_default config:

sed -i 's/^git_repo=.*$/git_repo=https:\/\/github.com\/ceph\/ceph.git/g' /data/sh/build.sh
sed -i 's/^git_branch_default=.*$/git_branch_default=pacific/g' /data/sh/build.sh

Download and build ceph:

cd /data/sh/
./build.sh

⁠Package Ceph

Set git_repo and git_branch_default config:

sed -i 's/^git_repo=.*$/git_repo=https:\/\/github.com\/ceph\/ceph.git/g' /data/sh/package.sh
sed -i 's/^git_branch_default=.*$/git_branch_default=pacific/g' /data/sh/package.sh

Download and package ceph:

cd /data/sh/
./package.sh

⁠Deploy Ceph

The relationship between ceph-ansible and pyenv:

Download ceph-ansible and install deps:

cd /data/code/
git clone https://github.com/ceph/ceph-ansible.git
cd ceph-ansible
git checkout stable-6.0
pyenv activate ceph-ansible-stable-6.0
pip install -r requirements.txt
ansible-galaxy install -r requirements.yml
pyenv deactivate

Init ceph topo config.

...

Deploy ceph:

cd /data/code/ceph-ansible/
pyenv activate ceph-ansible-stable-6.0

# Deploy
ansible-playbook -vvvv -i hosts site.yml

# Destroy and clean up
ansible-playbook -vvvv -i hosts infrastructure-playbooks/purge-cluster.yml

pyenv deactivate

⁠Others

⁠Start rpm web server

Check the rpm data directory and metadata index:

ls -al /data/web/ceph/
ls -al /data/web/ceph/RPMS/noarch/repodata/
ls -al /data/web/ceph/RPMS/x86_64/repodata/

Set ceph-custom.repo config:

sed -i 's/ipaddress/10.22.33.44/g' /data/web/ceph-custom.repo

Start nginx web server:

nginx

⁠Extra Sections

▶ Dockerfile

FROM centos:8.4.2105
COPY files /data
RUN rm -rf /etc/yum.repos.d \
    && cp -rf /data/conf/yum.repos.d /etc/yum.repos.d \
    && dnf clean all \
    && dnf groupinstall -y 'development tools' \
    && dnf install -y epel-release kernel-headers zlib zlib-devel bzip2-devel openssl-devel \
       sqlite-devel readline-devel python2-devel python36-devel python38-devel python2-jinja2 \
       python3-jinja2 python38-jinja2 git bc doxygen nginx snappy csnappy csnappy-devel libffi \
       libffi-devel sshpass createrepo jsonnet glibc-all-langpacks wget xz-devel \
    && dnf reinstall -y kernel-headers glibc-headers csnappy-devel snappy csnappy-devel \
       rpm-build rpmdevtools
RUN rm -rf /etc/nginx/nginx.conf /etc/ssh/sshd_config /root/.bashrc /root/.ssh/* \
    && cp /data/conf/nginx/nginx.conf /etc/nginx/nginx.conf \
    && cp /data/conf/ssh/sshd_config /etc/ssh/sshd_config \
    && cp /data/conf/bash/bashrc /root/.bashrc \
    && ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa \
    && rm -f /etc/localtime \
    && ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash \
    && source /root/.bashrc \
    && mkdir -p /root/.pyenv/cache/ \
    && wget https://npm.taobao.org/mirrors/python/3.6.15/Python-3.6.15.tar.xz -O /root/.pyenv/cache/Python-3.6.15.tar.xz \
    && wget https://npm.taobao.org/mirrors/python/3.8.16/Python-3.8.16.tar.xz -O /root/.pyenv/cache/Python-3.8.16.tar.xz \
    && pyenv install 3.6.15 3.8.16 \
    && pyenv global 3.8.16 \
    && pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple \
    && pip config set install.trusted-host https://pypi.tuna.tsinghua.edu.cn \
    && pyenv virtualenv ceph-pacific \
    && pyenv activate ceph-pacific \
    && pip install pip setuptools wheel Cython IPy boto boto3 xmltodict jwt prettytable --upgrade \
    && pyenv deactivate \
    && pyenv virtualenv ceph-quincy \
    && pyenv activate ceph-quincy \
    && pip install pip setuptools wheel Cython --upgrade \
    && pyenv deactivate \
    && pyenv virtualenv ceph-reef \
    && pyenv activate ceph-reef \
    && pip install pip setuptools wheel Cython --upgrade \
    && pyenv deactivate \
    && pyenv virtualenv ceph-ansible-stable-6.0 \
    && pyenv activate ceph-ansible-stable-6.0 \
    && pip install pip setuptools wheel Cython --upgrade \
    && pip install jinja2==3.0.3 \
    && wget https://github.com/ansible/ansible/archive/v2.10.7.tar.gz -O /tmp/Ansible-v2.10.7.tar.gz \
    && pip install /tmp/Ansible-v2.10.7.tar.gz \
    && pyenv deactivate \
    && pyenv virtualenv ceph-ansible-stable-7.0 \
    && pyenv activate ceph-ansible-stable-7.0 \
    && pip install pip setuptools wheel Cython --upgrade \
    && wget https://github.com/ansible/ansible/archive/v2.12.10.tar.gz -O /tmp/Ansible-v2.12.10.tar.gz \
    && pip install /tmp/Ansible-v2.12.10.tar.gz \
    && pyenv deactivate \
    && rm -rf /root/.pyenv/cache/Python-*.tar.xz /tmp/Ansible-*
RUN source /root/.bashrc \
    && git clone https://github.com/ceph/ceph-ansible.git /data/code/ceph-ansible \
    && cd /data/code/ceph-ansible \
    && git checkout stable-6.0 \
    && mkdir -p /root/ansible \
    && pyenv activate ceph-ansible-stable-6.0 \
    && pip install -r requirements.txt \
    && ansible-galaxy install -r requirements.yml \
    && pyenv deactivate

▶ conf/bash/bashrc

# .bashrc

# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# PyEnv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

delproxy() {
    proxies=$(git config --global --get-regexp '^url.*insteadof$' | awk -F ' ' '{print $1}')
    if test ! -z "$proxies"; then
        for proxy in "${proxies[@]}"; do
            key=$proxy
            val=$(git config --global --get $key)
            git config --global --unset $key
            echo "Delete github proxy: $key $val"
        done
    fi
}

lsproxy() {
    echo "Cmd: gitnp, Desc: disable github proxy"
    echo "Cmd: gitp1, Desc: enable github proxy with https://ghproxy.com/"
    echo "Cmd: gitp2, Desc: enable github proxy with https://gitclone.com/"
}

# disable github proxy
gitnp() {
    delproxy
}

# enable github proxy with https://ghproxy.com/
gitp1() {
    delproxy
    git config --global url."https://ghproxy.com/https://github.com/".insteadOf https://github.com/
    echo "Add github proxy: $(git config --global --get-regexp '^url.*insteadof$')"
}

# enable github proxy with https://gitclone.com/
gitp2() {
    delproxy
    git config --global url."https://gitclone.com/github.com/".insteadOf https://github.com/
    echo "Add github proxy: $(git config --global --get-regexp '^url.*insteadof$')"
}

# trim hosts
unalias cp
cp -f /etc/hosts /tmp/hosts
sed -i 's/^::1.*$/::1 ip6-localhost ip6-loopback/g' /tmp/hosts
cp -f /tmp/hosts /etc/hosts
alias cp="cp -i"

▶ conf/nginx/nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main    '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
    access_log          /var/log/nginx/access.log  main;
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    include             /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        autoindex    on;
        root         /data/web;
        include      /etc/nginx/default.d/*.conf;

        location / {
        }

        location ~* \.(repo|json|xml|yml|yaml|log)$ {
            add_header Content-Type "text/plain";
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

▶ conf/ssh/sshd_config

Include /etc/ssh/ssh_config.d/*.conf
Host *
    StrictHostKeyChecking no

▶ conf/yum.repos.d/centos-debuginfo.repo

[Debuginfo]
name=CentOS-$releasever - Debuginfo - tsinghua and aliyun
baseurl=https://mirrors.aliyun.com/centos-debuginfo/$releasever/$basearch/
enabled=0
gpgcheck=0

▶ conf/yum.repos.d/centos-epel.repo

[EPEL]
name=CentOS-$releasever - EPEL - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/$releasever/Everything/$basearch
        https://mirrors.aliyun.com/epel/$releasever/Everything/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir
enabled=1
gpgcheck=0

[EPEL-Debug]
name=CentOS-$releasever - EPEL - Debug - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/$releasever/Everything/$basearch/debug
        https://mirrors.aliyun.com/epel/$releasever/Everything/$basearch/debug
enabled=0
gpgcheck=0

[EPEL-Source]
name=CentOS-$releasever - EPEL - Source - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/$releasever/Everything/SRPMS
        https://mirrors.aliyun.com/epel/$releasever/Everything/SRPMS
enabled=0
gpgcheck=0

[EPEL-Modular]
name=CentOS-$releasever - EPEL - Modular(DEPRECATED) - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/$releasever/Modular
        https://mirrors.aliyun.com/epel/$releasever/Modular
enabled=0
gpgcheck=0

[EPEL-Modular-Debug]
name=CentOS-$releasever - EPEL - ModularDebug(DEPRECATED) - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/$releasever/Modular/$basearch/debug
        https://mirrors.aliyun.com/epel/$releasever/Modular/$basearch/debug
enabled=0
gpgcheck=0

[EPEL-Modular-Source]
name=CentOS-$releasever - EPEL - ModularSource(DEPRECATED) - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/$releasever/Modular/source/tree/
        https://mirrors.aliyun.com/epel/$releasever/Modular/source/tree/
enabled=0
gpgcheck=0

▶ conf/yum.repos.d/centos-stream.repo

[Stream-Base]
name=CentOS-$releasever - Stream - Base - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/BaseOS/$basearch/os/
        https://mirrors.aliyun.com/centos/8-stream/BaseOS/$basearch/os/
enabled=0
gpgcheck=0

[Stream-Extras]
name=CentOS-$releasever - Stream - Extras - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/extras/$basearch/os/
        https://mirrors.aliyun.com/centos/8-stream/extras/$basearch/os/
enabled=0
gpgcheck=0

[Stream-PowerTools]
name=CentOS-$releasever - Stream - PowerTools - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/PowerTools/$basearch/os/
        https://mirrors.aliyun.com/centos/8-stream/PowerTools/$basearch/os/
enabled=0
gpgcheck=0

[Stream-AppStream]
name=CentOS-$releasever - Stream - AppStream - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/AppStream/$basearch/os/
        https://mirrors.aliyun.com/centos/8-stream/AppStream/$basearch/os/
enabled=0
gpgcheck=0

[Stream-HighAvailability]
name=CentOS-$releasever - Stream - HighAvailability - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/HighAvailability/$basearch/os/
        https://mirrors.aliyun.com/centos/8-stream/HighAvailability/$basearch/os/
enabled=0
gpgcheck=0

[Stream-Plus]
name=CentOS-$releasever - Stream - Plus - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/centosplus/$basearch/os/
        https://mirrors.aliyun.com/centos/8-stream/centosplus/$basearch/os/
enabled=0
gpgcheck=0

[Stream-Devel]
name=CentOS-$releasever - Stream - Devel - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/Devel/$basearch/os/
        https://mirrors.aliyun.com/centos/8-stream/Devel/$basearch/os/
enabled=0
gpgcheck=0

▶ conf/yum.repos.d/centos-vault.repo

[Vault-Base]
name=CentOS-$releasever - Vault - Base - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/BaseOS/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/BaseOS/$basearch/os/
enabled=1
gpgcheck=0

[Vault-Extras]
name=CentOS-$releasever - Vault - Extras - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/extras/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/extras/$basearch/os/
enabled=1
gpgcheck=0

[Vault-PowerTools]
name=CentOS-$releasever - Vault - PowerTools - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/PowerTools/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/PowerTools/$basearch/os/
enabled=1
gpgcheck=0

[Vault-AppStream]
name=CentOS-$releasever - Vault - AppStream - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/AppStream/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/AppStream/$basearch/os/
enabled=1
gpgcheck=0

[Vault-HighAvailability]
name=CentOS-$releasever - Vault - HighAvailability - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/HighAvailability/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/HighAvailability/$basearch/os/
enabled=1
gpgcheck=0

[Vault-Plus]
name=CentOS-$releasever - Vault - Plus - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/centosplus/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/centosplus/$basearch/os/
enabled=0
gpgcheck=0

[Vault-Devel]
name=CentOS-$releasever - Vault - Devel - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/Devel/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/Devel/$basearch/os/
enabled=0
gpgcheck=0

[Vault-FastTrack]
name=CentOS-$releasever - Vault - FastTrack - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/fasttrack/$basearch/os/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/fasttrack/$basearch/os/
enabled=0
gpgcheck=0

▶ conf/yum.repos.d/ceph-pacific.repo

[Ceph]
name=Ceph packages for pacific - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-pacific/el8/$basearch
        https://mirrors.aliyun.com/ceph/rpm-pacific/el8/$basearch
enabled=1
gpgcheck=0
priority=99

[Ceph-noarch]
name=Ceph noarch packages for pacific - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-pacific/el8/noarch
        https://mirrors.aliyun.com/ceph/rpm-pacific/el8/noarch
enabled=1
gpgcheck=0
priority=99

[Ceph-source]
name=Ceph source packages for pacific - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-pacific/el8/SRPMS
        https://mirrors.aliyun.com/ceph/rpm-pacific/el8/SRPMS
enabled=0
gpgcheck=0
priority=99

[Ceph-packages]
name=Ceph packages for pacific - tsinghua and aliyun
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/8.4.2105/storage/$basearch/ceph-pacific/
        https://mirrors.aliyun.com/centos-vault/8.4.2105/storage/$basearch/ceph-pacific/
enabled=1
gpgcheck=0
priority=99

▶ sh/build.sh

#!/bin/bash -e

root_dir=/data
ceph_dir=$root_dir/code/ceph
pyenv_name=ceph-pacific
cpu_num=$(grep </proc/cpuinfo -c "processor")

# Custom conf
git_repo=
git_branch_default=
git_branch=${1:-"$git_branch_default"}

# Download ceph
if [ ! -d "$ceph_dir" ] || [ ! -d "$ceph_dir/.git" ]; then
    rm -rf $ceph_dir
    git clone "$git_repo" $ceph_dir
fi
cd $ceph_dir
git checkout -f "$git_branch"
git submodule update --init --recursive

# Build ceph
eval "$(pyenv init -)"
pyenv activate "$pyenv_name"
./install-deps.sh
./do_cmake.sh
cd build
make -j "$cpu_num"
pyenv deactivate

▶ sh/unittest.sh

#!/bin/bash -e

root_dir=/data
ceph_dir=$root_dir/code/ceph
pyenv_name=ceph-pacific

# Custom conf
git_repo=
git_branch_default=
git_branch=${1:-"$git_branch_default"}

# Download ceph
if [ ! -d "$ceph_dir" ] || [ ! -d "$ceph_dir/.git" ]; then
    rm -rf $ceph_dir
    git clone "$git_repo" $ceph_dir
fi
cd $ceph_dir
git checkout -f "$git_branch"
git submodule update --init --recursive

# Unittest ceph
eval "$(pyenv init -)"
pyenv activate "$pyenv_name"
./install-deps.sh
rm -rf build
./run-make-check.sh
pyenv deactivate

▶ sh/package.sh

#!/bin/bash -e

root_dir=/data
ceph_dir=$root_dir/code/ceph
rpm_dir=$root_dir/web/ceph
ceph_rpm_dir=$root_dir/ceph/rpmbuild
pyenv_name=ceph-pacific

# Custom conf
git_repo=
git_branch_default=
git_branch=${1:-"$git_branch_default"}

# Download ceph
if [ ! -d "$ceph_dir" ] || [ ! -d "$ceph_dir/.git" ]; then
    rm -rf $ceph_dir
    git clone "$git_repo" $ceph_dir
fi
cd $ceph_dir
git checkout -f "$git_branch"
git submodule update --init --recursive

# Package ceph
eval "$(pyenv init -)"
pyenv activate $pyenv_name
./install-deps.sh
./make-dist
mkdir -p $ceph_rpm_dir/SOURCES
cp ceph-*.tar.bz2 $ceph_rpm_dir/SOURCES
rpmbuild -ba --clean --rmsource --rmspec \
         --define="_topdir $ceph_rpm_dir" \
         --define="_rpmdir $ceph_rpm_dir/RPMS" \
         --define="_builddir $ceph_rpm_dir/BUILD" \
         --define="_sourcedir $ceph_rpm_dir/SOURCES" \
         --define="_specdir $ceph_rpm_dir/SPECS" \
         --define="_srcrpmdir $ceph_rpm_dir/SRPMS" \
         --define="_unpackaged_files_terminate_build 0" \
         ceph.spec --without selinux | tee rpm-build.log
pyenv deactivate

# Create rpm repo metadata
rm -rf /data/web/ceph
cp -rf $ceph_rpm_dir $rpm_dir
createrepo $rpm_dir/RPMS/x86_64/
createrepo $rpm_dir/RPMS/noarch/

▶ web/ceph-custom.repo

[Ceph-custom-stable]
name=Ceph custom stable $basearch repo
baseurl=http://ipaddress/ceph/RPMS/$basearch
enabled=1
gpgcheck=0
priority=0

[Ceph-custom-stable-noarch]
name=Ceph custom stable noarch repo
baseurl=http://ipaddress/ceph/RPMS/noarch
enabled=1
gpgcheck=0
priority=0

[Ceph-custom-stable-deps]
name=Ceph custom stable deps repo
baseurl=http://ipaddress/ceph/RPMS/deps
enabled=0
gpgcheck=0
priority=0

Tag summary

Recent tags

Content type

Image

Digest

sha256:2ab1559d1…

Size

826.5 MB

Last updated

over 3 years ago

docker pull bugwz/ceph-dev