현재 Kubernetes는 RHEL Little-endian(ppc64le)에서 지원됩니다.
기본 과정은 아래 블로그를 참조했습니다.
(https://www.ibm.com/developerworks/library/l-docker-orchestration/
http://cloudgeekz.com/773/setting-up-a-kubernetes-cluster-on-power.html
https://www.ibm.com/developerworks/community/blogs/mhhaque/entry/Docker_And_Kubernetes_Cluster_on_Power_with_RHEL7_Part_1_Preparing_all_Node?lang=en)
1. RHEL 7.2 환경에서 구성하며, kubernetes 설치를 위해 repository를 설정합니다.
[root@sys-88227 ~]# cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.2 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.2"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.2 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.2:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.2"
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Red Hat Enterprise Linux Server release 7.2 (Maipo)
=> kubernetes 설치하기 위한 파일셋 갖고 있는 repository를 추가
[root@sys-88227 ~]# cat > /etc/yum.repos.d/unicamp-docker.repo <<EOF
> [unicamp-docker]
> name=Unicamp Repo for Docker Packages
> baseurl=http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/
> enabled=1
> gpgcheck=0
> EOF
[root@sys-88227 ~]# cat > /etc/yum.repos.d/unicamp-misc.repo <<EOF
> [unicamp-misc]
> name=Unicamp Repo for Misc Packages
> baseurl=http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/misc_ppc64el/
> enabled=1
> gpgcheck=0
> EOF
=> 새로운 repository 추가된 것 확인
[root@sys-88227 ~]# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use su bscription-manager to register.
unicamp-docker | 2.9 kB 00:00
unicamp-misc | 2.9 kB 00:00
(1/2): unicamp-docker/primary_db | 3.9 kB 00:00
(2/2): unicamp-misc/primary_db | 21 kB 00:00
repo id repo name status
advance_toolchain IBM Advance Toolchain 88
compiler IBM XL C/C++ And Fortran Compiler 56
optional RedHat Enterprise LE Linux 7.2 - Optional 25,873
server RedHat Enterprise LE Linux 7.2 - Server 6,594
supplementary RedHat Enterprise LE Linux 7.2 - Supplementary 19,380
unicamp-docker Unicamp Repo for Docker Packages 3
unicamp-misc Unicamp Repo for Misc Packages 40
repolist: 52,034
=> Kubernetes 관련 파일셋 설치
[root@sys-88227 ~]# yum install kubernetes-client kubernetes-master etcd
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-man ager to register.
Resolving Dependencies
--> Running transaction check
---> Package etcd.ppc64le 0:3.0.15-1.el7 will be installed
---> Package kubernetes-client.ppc64le 0:1.4.7-1.git92b4f97.el7 will be installed
---> Package kubernetes-master.ppc64le 0:1.4.7-1.git92b4f97.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================
Package Arch Version Repository Size
==============================================================================================
Installing:
etcd ppc64le 3.0.15-1.el7 unicamp-misc 7.2 M
kubernetes-client ppc64le 1.4.7-1.git92b4f97.el7 unicamp-misc 6.6 M
kubernetes-master ppc64le 1.4.7-1.git92b4f97.el7 unicamp-misc 27 M
Transaction Summary
==============================================================================================
Install 3 Packages
Total download size: 41 M
Installed size: 270 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): kubernetes-client-1.4.7-1.git92b4f97.el7.ppc64le.rpm | 6.6 MB 00:00:09
(2/3): etcd-3.0.15-1.el7.ppc64le.rpm | 7.2 MB 00:00:10
(3/3): kubernetes-master-1.4.7-1.git92b4f97.el7.ppc64le.rpm | 27 MB 00:00:40
---------------------------------------------------------------------------------------------------------------
Total 830 kB/s | 41 MB 00:00:50
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kubernetes-client-1.4.7-1.git92b4f97.el7.ppc64le 1/3
groupadd: failure while writing changes to /etc/group
useradd: group 'kube' does not exist
error: %pre(kubernetes-master-1.4.7-1.git92b4f97.el7.ppc64le) scriptlet failed, exit status 6
Error in PREIN scriptlet in rpm package kubernetes-master-1.4.7-1.git92b4f97.el7.ppc64le
error: kubernetes-master-1.4.7-1.git92b4f97.el7.ppc64le: install failed
groupadd: failure while writing changes to /etc/group
useradd: group 'etcd' does not exist
error: %pre(etcd-3.0.15-1.el7.ppc64le) scriptlet failed, exit status 6
Error in PREIN scriptlet in rpm package etcd-3.0.15-1.el7.ppc64le
Verifying : kubernetes-client-1.4.7-1.git92b4f97.el7.ppc64le 1/3
Verifying : etcd-3.0.15-1.el7.ppc64le 2/3
Verifying : kubernetes-master-1.4.7-1.git92b4f97.el7.ppc64le 3/3
Installed:
kubernetes-client.ppc64le 0:1.4.7-1.git92b4f97.el7
Failed:
etcd.ppc64le 0:3.0.15-1.el7 kubernetes-master.ppc64le 0:1.4.7-1.git92b4f97.el7
Complete!
=> 그룹 추가 관련 명령어가 수행되지 않아서 Kubernetes-master, etcd 설치 실패.
=> 임의의 그룹 번호(700, 701) 사용 중인지 확인 후, 그룹 추가합니다.
[root@sys-88227 ~]# cat /etc/group | grep 700
[root@sys-88227 ~]# cat /etc/group | grep 701
[root@sys-88227 ~]# groupadd -g 700 kube
[root@sys-88227 ~]# groupadd -g 701 etcd
[root@sys-88227 ~]# cat /etc/group | grep kube
kube:x:700:
[root@sys-88227 ~]# cat /etc/group | grep etcd
etcd:x:701:
=> Kubernetes-master, etcd 설치 재수행
[root@sys-88227 ~]# yum install kubernetes-master etcd
(중략)
Installed:
etcd.ppc64le 0:3.0.15-1.el7 kubernetes-master.ppc64le 0:1.4.7-1.git92b4f97.el7
Complete!
,
=> Network Ports 오픈 (Kubernetes API 서버 : 8080, etcd : 2379 by default)
[root@sys-88227 ~]# firewall-cmd --zone=public --add-port 8080/tcp --permanent
success
[root@sys-88227 ~]# firewall-cmd --zone=public --add-port=2379/tcp --permanent
success
[root@sys-88227 ~]# firewall-cmd --reload
success
2. Kubernetes Master 구성
설치가 되었으면 아래와 같이 /etc/kubernetes 폴더가 생깁니다.
[root@sys-88227 ~]# ls /etc/kubernetes/
apiserver config controller-manager scheduler
=> Master로 구성할 서버의 IP 정보 확인
[root@sys-88227 ~]# ifconfig | grep inet
inet 172.29.160.107 netmask 255.255.192.0 broadcast 172.29.191.255
[root@sys-88227 ~]# openssl genrsa -out /tmp/serviceaccount.key 2048
=> Kubernetes 구성 파일에서 Master IP 정보 수정
[root@sys-88227 ~]# vi /etc/kubernetes/config
# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://172.29.160.107:8080"
=> API 서버 구성파일에서 etcd 서버의 IP 정보 수정 (현재 구성에서는 One Master)
[root@sys-88227 ~]# vi /etc/kubernetes/apiserver
# The address on the local server to listen to.
KUBE_API_ADDRESS="--address=0.0.0.0"
# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://172.29.160.107:2379,http://172.29.160.107:4001"
KUBE_API_PORT="--port=8080"
KUBE_API_ARGS="--service_account_key_file=/tmp/serviceaccount.key"
=> Etcd 구성하기
[root@sys-88227 ~]# vi /etc/etcd/etcd.conf
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
=> Control-manager 추가
KUBE_CONTROLLER_MANAGER_ARGS="--service_account_private_key_file=/tmp/serviceaccount.key"
=> Kubernetes 서비스 시작하기
[root@sys-88227 ~]# for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
> systemctl restart $SERVICES
> systemctl enable $SERVICES
> systemctl status $SERVICES
> done
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-07-16 11:06:08 EDT; 214ms ago
Main PID: 9538 (etcd)
CGroup: /system.slice/etcd.service
└─9538 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://0.0.0.0:23...
Jul 16 11:06:07 sys-88227.dal-ebis.ihost.com etcd[9538]: starting server... [version: 3.0.15, cluster version: 3.0]
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: 8e9e05c52164694d is starting a new election at term 5
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: 8e9e05c52164694d became candidate at term 6
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: 8e9e05c52164694d received vote from 8e9e05c52164694d at term 6
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: 8e9e05c52164694d became leader at term 6
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 6
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: published {Name:default ClientURLs:[http://0.0.0.0:2379]} to c...8c32
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: ready to serve client requests
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com etcd[9538]: serving insecure client requests on 0.0.0.0:2379, this is stro...ged!
Jul 16 11:06:08 sys-88227.dal-ebis.ihost.com systemd[1]: Started Etcd Server.
Hint: Some lines were ellipsized, use -l to show in full.
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-07-16 11:06:09 EDT; 86ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 9571 (kube-apiserver)
CGroup: /system.slice/kube-apiserver.service
└─9571 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://172.29.160.107:2379 --insecure-bin...
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: E0716 11:06:09.031808 9571 reflector.go:214] k8s.i...sed
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: E0716 11:06:09.031936 9571 reflector.go:214] k8s.io/k...
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: E0716 11:06:09.032020 9571 reflector.go:214] k8s.i...sed
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: E0716 11:06:09.032143 9571 reflector.go:203] k8s.i...sed
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: E0716 11:06:09.139549 9571 reflector.go:214] pkg/c...sed
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: [restful] 2017/07/16 11:06:09 log.go:30: [restful/swa...pi/
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: [restful] 2017/07/16 11:06:09 log.go:30: [restful/swa...ui/
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com systemd[1]: Started Kubernetes API Server.
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: I0716 11:06:09.257313 9571 genericapiserver.go:716...443
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com kube-apiserver[9571]: I0716 11:06:09.257490 9571 genericapiserver.go:761...080
Hint: Some lines were ellipsized, use -l to show in full.
● kube-controller-manager.service - Kubernetes Controller Manager
Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-07-16 11:06:09 EDT; 148ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 9599 (kube-controller)
CGroup: /system.slice/kube-controller-manager.service
└─9599 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://172.29.160.107:8080 --service_a...
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com systemd[1]: Stopping Kubernetes Controller Manager...
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com systemd[1]: Started Kubernetes Controller Manager.
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com systemd[1]: Starting Kubernetes Controller Manager...
● kube-scheduler.service - Kubernetes Scheduler Plugin
Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-07-16 11:06:09 EDT; 144ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 9626 (kube-scheduler)
CGroup: /system.slice/kube-scheduler.service
└─9626 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://172.29.160.107:8080
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com systemd[1]: Stopping Kubernetes Scheduler Plugin...
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com systemd[1]: Started Kubernetes Scheduler Plugin.
Jul 16 11:06:09 sys-88227.dal-ebis.ihost.com systemd[1]: Starting Kubernetes Scheduler Plugin...
3. Kubernetes node (Minion) 구성하기
=> 필요한 패키지 설치
[root@sys-88228 ~]#yum install kubernetes-client kubernetes-node
1:nvidia-kmod-361.107-2.el7.ppc64le has installed conflicts nvidia-uvm-kmod: 1:nvidia-kmod-361.107-2.el7.ppc64le
Installing : kubernetes-client-1.4.7-1.git92b4f97.el7.ppc64le 1/3
Installing : socat-1.7.2.2-5.el7.ppc64le 2/3
Installing : kubernetes-node-1.4.7-1.git92b4f97.el7.ppc64le 3/3
Verifying : kubernetes-node-1.4.7-1.git92b4f97.el7.ppc64le 1/3
Verifying : socat-1.7.2.2-5.el7.ppc64le 2/3
Verifying : kubernetes-client-1.4.7-1.git92b4f97.el7.ppc64le 3/3
Installed:
kubernetes-client.ppc64le 0:1.4.7-1.git92b4f97.el7 kubernetes-node.ppc64le 0:1.4.7-1.git92b4f97.el7
Dependency Installed:
socat.ppc64le 0:1.7.2.2-5.el7
Complete!
[root@sys-88228 ~]# vi /etc/kubernetes/config
# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://172.29.160.107:8080"
[root@sys-88228 ~]# vi /etc/kubernetes/kubelet
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME=" "
# location of the api-server
KUBELET_API_SERVER="--api-servers=http://172.29.160.107:8080"
[root@sys-88228 ~]# for SERVICES in kube-proxy kubelet docker; do
> systemctl restart $SERVICES
> systemctl enable $SERVICES
> systemctl status $SERVICES
> done
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
● kube-proxy.service - Kubernetes Kube-Proxy Server
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-07-13 22:40:11 EDT; 177ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 21334 (kube-proxy)
CGroup: /system.slice/kube-proxy.service
└─21334 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=h...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: I0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: W0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: I0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: I0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: I0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: I0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: I0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: E0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: E0713 22:4...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kube-proxy[21334]: E0713 22:4...
Hint: Some lines were ellipsized, use -l to show in full.
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-07-13 22:40:11 EDT; 269ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 21374 (kubelet)
CGroup: /system.slice/kubelet.service
└─21374 /usr/bin/kubelet --logtostderr=true --v=0 --api-servers...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com systemd[1]: Started Kubernete...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com systemd[1]: Starting Kubernet...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kubelet[21374]: Flag --api-se...
Jul 13 22:40:11 sys-88228.dal-ebis.ihost.com kubelet[21374]: W0713 22:40:1...
Hint: Some lines were ellipsized, use -l to show in full.
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-07-13 22:40:14 EDT; 215ms ago
Docs: https://docs.docker.com
Main PID: 21403 (dockerd)
CGroup: /system.slice/docker.service
├─21403 /usr/bin/dockerd
└─21407 docker-containerd -l unix:///var/run/docker/libcontaine...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com dockerd[21403]: time="2017-07...
Jul 13 22:40:14 sys-88228.dal-ebis.ihost.com systemd[1]: Started Docker Ap...
Hint: Some lines were ellipsized, use -l to show in full.
4. Master 서버에서 node 정보 확인
[root@sys-88227 ~]# kubectl get nodes --show-labels
NAME STATUS AGE LABELS
127.0.0.1 NotReady 2d beta.kubernetes.io/arch=ppc64le,beta.kubernetes.io/os=linux,kubernetes.io/hostname=127.0.0.1
sys-88228.dal-ebis.ihost.com Ready 11m beta.kubernetes.io/arch=ppc64le,beta.kubernetes.io/os=linux,kubernetes.io/hostname=sys-88228.dal-ebis.ihost.com
