Run a full-fledged cluster on Kubernetes from scratch on Ubuntu 16.04
Already quite a lot of articles have been written on installing and launching Kubernetes , however, not everything is so smooth (I spent several days launching my cluster).
This article is intended to provide comprehensive information not only on installing k8s, but also to explain each step: why and why we are doing exactly as it is written (this is very important for a successful launch).
Servers:
A cluster means that you have more than one physical server between which resources will be distributed. Servers are called nodes.
')
Disks:
Normal hards in k8s are not supported. Work with disks occurs by means of distributed file storages. This is necessary so that k8s can “move” docker containers to other nodes, if necessary, without losing data (files).
You need to start creating a cluster by creating your own distributed file storage. If you are sure that you will never need disks, you can skip this step.
I chose Ceph . And I recommend reading this wonderful article .
The minimum reasonable number of servers for Ceph is 3 (you can build on one, but this makes little sense because of the high probability of losing data).
Network:
We need Flannel - it allows you to organize a software defined network (Software Defined Network, SDN). It is SDN that allows all our containers to communicate with each other within the cluster (the Flannel installation is done with k8s and described below).
In our example, we use 3 physical servers. Install Ubuntu 16.04 on all servers. Do not create swap partitions (k8s requirement).
Provide at least one disk (or partition) for Ceph in each server.
Do not enable SELinux support (it is turned off by default in Ubuntu 16.04).
We called the server like this: kub01 kub02 kub03. The sda2 part on each server is created for Ceph (formatting is optional).
I will describe the Ceph installation fairly briefly. There are many examples on the web and the Ceph site itself has pretty good documentation.
We perform all operations from under the privileged root user.
Create a temporary directory:
Install Ceph:
You need to create a key and decompose it across all servers. This is needed for the ceph-deploy utility:
(you may need to correct the ssh config to allow it to log in as root).
Check that kub01 is not registered in your / etc / hosts as 127.0.0.1 (if registered, delete this line).
Create a disk cluster and initialize it:
Check our disk cluster:
You can use the following useful commands:
Now that we have verified that Ceph is working, we will create a separate pool for k8s:
(you can view all existing pools with: ceph df)
Now we will create a separate user for our kube pool and save the keys:
(you will need the keys to access the k8s storage)
Add the k8s repository to our system:
Now install the main packages:
Initialize and run k8s
(such a network 10.244.0.0/16 is necessary for the flannel to work - do not change it)
Save the script printed command to attach the nodes to the cluster.
It is convenient to use a separate unprivileged user for working with k8s. Create it and copy the k8s configuration file into it:
The utility is used to work with k8s: kubectl . We use it only from under our user kube . To go under the user run:
Allow containers to run on the wizard:
Configuring rights:
Install the flannel (network subsystem):
Create a user to access the web interface:
Run kube-proxy, you can do it like this:
And forward port 8001 from your working machine to kub01 server:
Now we can access the web interface from our working machine at:
http://127.0.0.1:8001/ui
(a web interface will open where you need to specify a token)
You can get a token to access the web interface like this:
The current kube-controller-manager-amd64 controller: v1.9.2 is missing the rbd binary required for working with Ceph , so we will create our own kube-controller-manager:
To create our kube-controller-manager, do the following:
(we execute all commands from under the root user)
(Be sure to include current versions of the k8s and the OS distribution)
We check that our controller is successfully created:
Check that our image has rbd:
Should see something like this: rbd: / usr / bin / rbd /usr/share/man/man8/rbd.8.gz
We replace the standard controller with ours, for this we edit the file:
/etc/kubernetes/manifests/kube-controller-manager.yaml
Replace the string:
image: gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2
on:
image: my-kube-controller-manager: v1.9.2
imagePullPolicy: IfNotPresent
(be sure to add the imagePullPolicy directive so that k8s does not try to download this image from the Internet)
We go under the user kube and wait until our controller starts up (nothing needs to be done).
Must see that our image is used:
Image: my-kube-controller-manager: v1.9.2
Now that our controller with RBD support has been launched, we can start setting up a bunch of k8s and Ceph.
Add keys to k8s to access Ceph:
Create StorageClass (default):
On the new server, run the following commands:
We attach the node to the master:
We need a key. You can get it on the wizard by running the command:
Or, create it:
An example of a command to join (execute on a new node):
For quick and easy installation of applications in k8s Helm was invented.
A list of available applications can be found here .
Install and initialize Helm:
PS: it took 6 hours to write this article. Do not judge strictly if there are typos somewhere. Ask questions, gladly answer and help.
This article is intended to provide comprehensive information not only on installing k8s, but also to explain each step: why and why we are doing exactly as it is written (this is very important for a successful launch).
What you need to know
Servers:
A cluster means that you have more than one physical server between which resources will be distributed. Servers are called nodes.
')
Disks:
Normal hards in k8s are not supported. Work with disks occurs by means of distributed file storages. This is necessary so that k8s can “move” docker containers to other nodes, if necessary, without losing data (files).
You need to start creating a cluster by creating your own distributed file storage. If you are sure that you will never need disks, you can skip this step.
I chose Ceph . And I recommend reading this wonderful article .
The minimum reasonable number of servers for Ceph is 3 (you can build on one, but this makes little sense because of the high probability of losing data).
Network:
We need Flannel - it allows you to organize a software defined network (Software Defined Network, SDN). It is SDN that allows all our containers to communicate with each other within the cluster (the Flannel installation is done with k8s and described below).
Server Preparation
In our example, we use 3 physical servers. Install Ubuntu 16.04 on all servers. Do not create swap partitions (k8s requirement).
Provide at least one disk (or partition) for Ceph in each server.
Do not enable SELinux support (it is turned off by default in Ubuntu 16.04).
We called the server like this: kub01 kub02 kub03. The sda2 part on each server is created for Ceph (formatting is optional).
Install and configure Ceph
I will describe the Ceph installation fairly briefly. There are many examples on the web and the Ceph site itself has pretty good documentation.
We perform all operations from under the privileged root user.
Create a temporary directory:
mkdir ~/ceph-admin cd ~/ceph-admin Install Ceph:
apt install ceph-deploy ceph-common
You need to create a key and decompose it across all servers. This is needed for the ceph-deploy utility:
ssh-keygen ssh-copy-id kub01 ssh-copy-id kub02 ssh-copy-id kub03
(you may need to correct the ssh config to allow it to log in as root).
Check that kub01 is not registered in your / etc / hosts as 127.0.0.1 (if registered, delete this line).
Create a disk cluster and initialize it:
ceph-deploy new kub01 kub02 kub03 ceph-deploy install kub01 kub02 kub03 ceph-deploy mon create-initial ceph-deploy osd prepare kub01:sda2 kub02:sda2 kub03:sda2 ceph-deploy osd activate kub01:sda2 kub02:sda2 kub03:sda2
Check our disk cluster:
ceph -s cluster 363a4cd8-4cb3-4955-96b2-73da72b63cf5 health HEALTH_OK
You can use the following useful commands:
ceph -s ceph df ceph osd tree
Now that we have verified that Ceph is working, we will create a separate pool for k8s:
ceph osd pool create kube 100 100
(you can view all existing pools with: ceph df)
Now we will create a separate user for our kube pool and save the keys:
ceph auth get-or-create client.kube mon 'allow r' osd 'allow rwx pool=kube' ceph auth get-key client.admin > /etc/ceph/client.admin ceph auth get-key client.kube > /etc/ceph/client.kube
(you will need the keys to access the k8s storage)
Install Kubernetes
Add the k8s repository to our system:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF
Now install the main packages:
apt update apt install -y docker.io kubelet kubeadm kubernetes-cni
Initialize and run k8s
kubeadm init --pod-network-cidr=10.244.0.0/16
(such a network 10.244.0.0/16 is necessary for the flannel to work - do not change it)
Save the script printed command to attach the nodes to the cluster.
It is convenient to use a separate unprivileged user for working with k8s. Create it and copy the k8s configuration file into it:
useradd -s /bin/bash -m kube mkdir ~kube/.kube cp /etc/kubernetes/admin.conf ~kube/.kube/config chown kube: ~kube/.kube/config
The utility is used to work with k8s: kubectl . We use it only from under our user kube . To go under the user run:
su - kube
Allow containers to run on the wizard:
kubectl taint nodes --all node-role.kubernetes.io/master-
Configuring rights:
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
Install the flannel (network subsystem):
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
How to check that everything works
Run the command:
The output should be something like:
kubectl -n kube-system get pods
The output should be something like:
NAME READY STATUS RESTARTS AGE etcd-kub01.domain.com 1/1 Running 1 4d kube-apiserver-kub01.domain.com 1/1 Running 1 4d kube-controller-manager-kub01.domain.com 1/1 Running 0 4d kube-dns-7c6d8859cb-dmqrn 3/3 Running 0 1d kube-flannel-ds-j948h 1/1 Running 0 1d kube-proxy-rmbqq 1/1 Running 0 1d kube-scheduler-kub01.domain.com 1/1 Running 1 4d
Install and configure the web interface
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
Create a user to access the web interface:
cat << EOF > account.yaml apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kube-system EOF kubectl -n kube-system create -f account.yaml
Run kube-proxy, you can do it like this:
kubectl proxy &
And forward port 8001 from your working machine to kub01 server:
ssh -L 8001:127.0.0.1:8001 -N kub01 &
Now we can access the web interface from our working machine at:
http://127.0.0.1:8001/ui
(a web interface will open where you need to specify a token)
You can get a token to access the web interface like this:
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Customize Kubernetes and Ceph
The current kube-controller-manager-amd64 controller: v1.9.2 is missing the rbd binary required for working with Ceph , so we will create our own kube-controller-manager:
What is RBD, why is it needed and why is it missing?
RBD (Rados Block Device) is a block device that is used by k8s to create and mount Docker container partitions. In this case, this is the binary included in the package: ceph-common .
k8s does not include this package in its controller apparently because it depends on the distribution of the operating system you are using. Therefore, when assembling your controller, be sure to specify your distribution exactly, so that RBD is relevant.
k8s does not include this package in its controller apparently because it depends on the distribution of the operating system you are using. Therefore, when assembling your controller, be sure to specify your distribution exactly, so that RBD is relevant.
To create our kube-controller-manager, do the following:
(we execute all commands from under the root user)
mkdir docker cat << EOF > docker/Dockerfile FROM ubuntu:16.04 ARG KUBERNETES_VERSION=v1.9.2 ENV DEBIAN_FRONTEND=noninteractive \ container=docker \ KUBERNETES_DOWNLOAD_ROOT=https://storage.googleapis.com/kubernetes-release/release/${KUBERNETES_VERSION}/bin/linux/amd64 \ KUBERNETES_COMPONENT=kube-controller-manager RUN set -x \ && apt-get update \ && apt-get install -y \ ceph-common \ curl \ && curl -L ${KUBERNETES_DOWNLOAD_ROOT}/${KUBERNETES_COMPONENT} -o /usr/bin/${KUBERNETES_COMPONENT} \ && chmod +x /usr/bin/${KUBERNETES_COMPONENT} \ && apt-get purge -y --auto-remove \ curl \ && rm -rf /var/lib/apt/lists/* EOF docker build -t "my-kube-controller-manager:v1.9.2" docker/
(Be sure to include current versions of the k8s and the OS distribution)
We check that our controller is successfully created:
docker images | grep my-kube-controller-manager
Check that our image has rbd:
docker run my-kube-controller-manager:v1.9.2 whereis rbd
Should see something like this: rbd: / usr / bin / rbd /usr/share/man/man8/rbd.8.gz
We replace the standard controller with ours, for this we edit the file:
/etc/kubernetes/manifests/kube-controller-manager.yaml
Replace the string:
image: gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2
on:
image: my-kube-controller-manager: v1.9.2
imagePullPolicy: IfNotPresent
(be sure to add the imagePullPolicy directive so that k8s does not try to download this image from the Internet)
We go under the user kube and wait until our controller starts up (nothing needs to be done).
kubectl -n kube-system describe pods | grep kube-controller
Must see that our image is used:
Image: my-kube-controller-manager: v1.9.2
Now that our controller with RBD support has been launched, we can start setting up a bunch of k8s and Ceph.
Setting up the disk subsystem bundle (k8s + Ceph)
Add keys to k8s to access Ceph:
kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" --from-file=/etc/ceph/client.admin --namespace=kube-system kubectl create secret generic ceph-secret-kube --type="kubernetes.io/rbd" --from-file=/etc/ceph/client.kube --namespace=default
Create StorageClass (default):
cat << EOF > ceph_storage.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-rbd annotations: {"storageclass.kubernetes.io/is-default-class":"true"} provisioner: kubernetes.io/rbd parameters: monitors: kub01:6789,kub02:6789,kub03:6789 pool: kube adminId: admin adminSecretName: ceph-secret adminSecretNamespace: "kube-system" userId: kube userSecretName: ceph-secret-kube fsType: ext4 imageFormat: "2" imageFeatures: "layering" EOF kubectl create -f ceph_storage.yaml
How to check the operation of the disk subsystem?
Checking the availability of StorageClass:
Create a test pod with a disk:
Check that the Pod was created (you need to wait for the creation and launch):
Check that Claim was created (request for a disc):
Check that the disk itself is created:
Finally, check that the drive is mounted on the system:
kube@kub01:~$ kubectl get storageclass NAME PROVISIONER AGE ceph-rbd (default) kubernetes.io/rbd 4d
Create a test pod with a disk:
cat << EOF > test_pod.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: claim1 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi --- apiVersion: v1 kind: Pod metadata: name: test-pod-with-pvc spec: volumes: - name: test-pvc-storage persistentVolumeClaim: claimName: claim1 containers: - name: test-container image: kubernetes/pause volumeMounts: - name: test-pvc-storage mountPath: /var/lib/www/html EOF kubectl create -f test_pod.yaml
Check that the Pod was created (you need to wait for the creation and launch):
kube@kub01:~$ kubectl get pods NAME READY STATUS RESTARTS AGE test-pod-with-pvc 1/1 Running 0 15m
Check that Claim was created (request for a disc):
kube@kub01:~$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE claim1 Bound pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b 1Gi RWO ceph-rbd 12m
Check that the disk itself is created:
kube@kub01:~$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b 1Gi RWO Delete Bound default/claim1 ceph-rbd
Finally, check that the drive is mounted on the system:
root@kub01:~$ mount | grep pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b /dev/rbd0 on /var/lib/kubelet/pods/076fff13-0ce9-11e8-8b93-901b0e8fc39b/volumes/kubernetes.io~rbd/pvc-076df6ee-0ce9-11e8-8b93-901b0e8fc39b type ext4 (rw,relatime,stripe=1024,data=ordered)
Adding new (additional) nodes to the k8s cluster
On the new server, run the following commands:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF apt update apt install -y docker.io kubelet kubeadm kubernetes-cni ceph-common python
We attach the node to the master:
We need a key. You can get it on the wizard by running the command:
kubeadm token list
Or, create it:
kubeadm token create --print-join-command
An example of a command to join (execute on a new node):
kubeadm join --token cb9141.6a912d1dd7f66ff5 8.8.8.8:6443 --discovery-token-ca-cert-hash sha256:f0ec6d8f9699169089c89112e0e6b5905b4e1b42db22815186240777970dc6fd
Install and configure Helm
For quick and easy installation of applications in k8s Helm was invented.
A list of available applications can be found here .
Install and initialize Helm:
curl https://storage.googleapis.com/kubernetes-helm/helm-v2.8.0-linux-amd64.tar.gz | tar -xz ./linux-amd64/helm init
PS: it took 6 hours to write this article. Do not judge strictly if there are typos somewhere. Ask questions, gladly answer and help.
Source: https://habr.com/ru/post/348688/
All Articles