My hardware also different, if you don’t have such powerful machine, it can skip numa part, that’s only for high end ARM64 machine.
The hardware is Ampere Mt. Collins which has 2*Ampere Altra 80 Cores CPU and 8*32G Memory, my experiment is run different VM on different CPU, to seperate them and lock it on different numa node.

There are two CPUs, so, it can create 3 nodes, so, I have plan to allocate resource like that.
1. k8s master on socket 0: 10 CPUs, 16G memory
2. k8s node1 on socket 0 : 70 CPUs, 100G memory
3. k8s node2 on socket 1: 80 CPUs, 120G memory

Kubernetes version v1.28.2
Ubuntu 22.04

Notice: Before reading this document, this document is personal experiment, not Ampere official document, take your own risk.

Prepare VM from libvirt

The server has remote control system which problem sufficent resolution to use GUI on server, so, I install ubuntu 22.04 on host system, than use virt-manager to create k8s master node, for qemu image, I create the disk image alone, seems create image when run install will allocate all disk space, seems that’s a virt-manager bug.
And the host network is configured to be a bridge interface, in my case, that’s br0.

So, when uses virt-manager to create VM, it can bridge network interface to br0.

Configure the VM with Numa

It can use numastat to check currnet numa status, this system has two numa nodes. it can use numastat and numactl to get more numa information.

# numastat -n

Per-node numastat info (in MBs):
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
Numa_Hit               591239.17       791075.41      1382314.58
Numa_Miss                   0.00            0.00            0.00
Numa_Foreign                0.00            0.00            0.00
Interleave_Hit              8.63            7.13           15.77
Local_Node             519434.17       777974.99      1297409.16
Other_Node              71805.00        13100.42        84905.42

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 128377 MB
node 0 free: 80224 MB
node 1 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 1 size: 126426 MB
node 1 free: 93888 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

virt-manager doesn’t have advance command to assign CPU, probably modify xml can get that purpose, but it more converince to use the “virsh” cli command to adjust VM parameters. “virsh list” show current VMs, “virsh vcpupin list current VM CPU scheduling”, “virsh vcpupin domain-id vcpu cpulist” to modify the VCPU run on which one CPU.

For memory, it needs to use “virsh numatune” to lock the memory on different node.
“virsh numatune domain [–mode mode] [–nodeset nodeset] [[–config] [–live] | [–current]]”, –current can update current running VM machine.

$ virsh list
 Id   Name          State
-----------------------------
 9    k8s-master   running
 10   k8s-node1     running

$ virsh vcpupin k8s-master
 VCPU   CPU Affinity
----------------------
 0      0-159
 1      0-159
 2      0-159
 3      0-159
 4      0-159
 5      0-159
 6      0-159
 7      0-159
 8      0-159
 9      0-159

# this command can lock 9 k8s-master's VCPU 0 on physical CPU 0. 
$ virsh vcpupin 9 0 0 

# Show current VM's numatune, for example, following is unset status. 
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   :

$ virsh numatune 9 --nodeset 0

# the numa_nodeset was set to 0
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   : 0

The problem is, seems this script need to run everytime, alternativate solution is to modify XML, but Mt. collis has 160 cores, I won’t to modify the XML file. script is more converience on this situation, following is my sample script to adjust k8s-master, k8s-node1, and k8s-node2’s parameter, it can easy to modify it if you install other system or AltraMax CPU.

#!/bin/bash

configure_vm(){
        local VM_NAME="$1"
        local VM_VCPU="$2"
        local VM_CPU="$3"
        local VM_NUMA="$4"
        local VM_TYPE="$5" # VM_TYPE for MASTER or NODE0, NODE1 .. self define name

        id=$(virsh list | grep $VM_NAME | awk '{print $1}')
        state=$(virsh list | grep $VM_NAME | awk '{print $3}')

        if [ "$state" != "running" ]; then
          echo "$VM_TYPE is not running"
          exit 1
        else
          echo "configure $VM_TYPE"
        fi
        numa_nodeset=$(virsh numatune $id | grep numa_nodeset | awk '{print $3}')

        start_cpu=$(echo $VM_VCPU | cut -d'-' -f1)
        end_cpu=$(echo $VM_VCPU | cut -d'-' -f2)
        for ((vcpu = start_cpu; vcpu <= end_cpu; vcpu++)); do
          # echo "$id $vcpu $VM_CPU"
          virsh vcpupin $id $vcpu $VM_CPU > /dev/null
        done
        if [ -n "$numa_nodeset" ]; then
          echo "$VM_TYPE numa_nodeset is $numa_nodeset"
        else
          # echo "$id $VM_NUMA"
          virsh numatune $id --nodeset $VM_NUMA
        fi

}

configure_vm "k8s-master" "0-9" "0-9" "0" "Master"
configure_vm "k8s-node1" "0-69" "10-79" "0" "Node0"
configure_vm "k8s-node2" "0-79" "80-159" "1" "Node1"


====

# After run the script, it can run numastat <pid> to see the qemu runs numa status. ex:
$ sudo numastat 206601

Per-node process memory usage (in MBs) for PID 206601 (qemu-system-aar)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                        11.51            0.00           11.51
Stack                        0.04            0.00            0.04
Private                   5984.93            0.00         5984.94
----------------  --------------- --------------- ---------------
Total                     5996.48            0.00         5996.48

In here, on socket 0 , 0-9 and 0-69 is vcpu, so, all start from 0. After run this script, it can use “virsh vcpupin” and “virsh numatune” to get current setting. Maybe cpuset parameter in XML can more easy to get this function, but I have not try it.

Install K8s

Here is my script, just reference some later blog and my previous configurtion script, so, it can copy and paste to system and run it directly.

Install basic system and k8s

# setup timezone
sudo apt install ntpdate -y
sudo ntpdate watch.stdtime.gov.tw
sudo timedatectl set-timezone Asia/Taipei

# install and enable docker

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update
sudo apt install -y containerd.io


containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

echo "----------------"
echo "   swapoff"
echo "----------------"
sudo swapoff -a
sudo sed -i '/swap/ s/^\(.*\)$/#\1/g' /etc/fstab

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update

sudo apt-get -y install kubelet
sudo apt-get -y install kubeadm
sudo apt-get -y install kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# Update kernel parametes
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# update sysctl
sudo sysctl --system

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

echo "if this is work node, it can run kubeadm join now"

echo "alias k=kubectl" >> .bashrc

Setup Master Node

Run this script on master node.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# echo "enable master to run pods"
# kubectl taint nodes --all node-role.kubernetes.io/master-

It will shows some message below, then run it on k8s-node1 and k8s-node2

kubeadm join 192.168.110.230:6443 –token e14f7k.35c3jpny81ry70re \

        –discovery-token-ca-cert-hash sha256:6636a7e4ec3542f19152f308f0789c52287b743b60efa3c13e7c23ce1147c581

If forgot command above, it can create another token to let other k8s node join, ex:

kubeadm token create --print-join-command

Flannel : Network fabric for container

K8s need a network fabric to provide internal network topology, there are many choices, but we will install matellb later, some other network like calico might have potential issue, so, we choice flannel as network fabric.

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

After run this command, the k8s node status will change to Ready, if not install network fabric, the node status will be NotReady.

$ kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   26h   v1.28.2
k8s-node1    Ready    <none>          26h   v1.28.2
k8s-node2    Ready    <none>          26h   v1.28.2

Helm: Kubbernetes Applications Manager

Install helm to further install metallb purpose, on previous article, helm2 needs another application to connect to k8s, but helm3 doesn’t need it, more easy to use on k8s.

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o /tmp/get_helm.sh
chmod 700 /tmp/get_helm.sh
sudo /tmp/get_helm.sh

Metallb: Bere metal load-balancer for K8s

Install k8s on private side, it needs load-balancer to provide external IP to communication with internet or intranet. If you install Metallb, the “kubectl get svc”, the EXTERNAL-IP shows “Pending”, and k8s doesn’t response the arp on host, probably it doesn’t configure kube-proxy, it you have experience on previous metallb installtion, it might miss this part.

PATH=$PATH:/usr/local/bin/

helm repo add metallb https://metallb.github.io/metallb
helm --namespace metallb-system install --create-namespace metallb metallb/metallb -f metallb-values.yaml

# https://metallb.universe.tf/installation/
# see what changes would be made, returns nonzero returncode if different
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl diff -f - -n kube-system

# actually apply the changes, returns nonzero returncode on errors only
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl apply -f - -n kube-system

metallb-values.yaml is also different like previous setting, it needs to configure L2Advertisement on metallb, without it, your metallb won’t work. Follwing is metallb-values.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: primary-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.110.200-192.168.110.219

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - primary-pool

Traefik: TBD

Linkerd

Just show the install script here

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

echo "export PATH=$PATH:/home/$USER/.linkerd2/bin" >> ~/.bashrc

linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -

Other

it can insert isolinux to host linux kernel parameter to isolate the Linux scheduling to k8s nodes. (TBD)

isolcpus=10-159 nohz=on nohz_full=10-159

Test with Nginx

It can deploy nginx to test this system

kubectl create deploy nginx --image=nginx
kubectl expose deploy nginx --port=80 --target-port=80 --type=LoadBalancer
kubectl get svc nginx
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
nginx   LoadBalancer   10.99.211.255   192.168.110.200   80:31011/TCP   25h

Now, it can try to access http://192.168.110.200.
if want to Delete nginx deploy, run following command.

kubectl delete deploy nginx
kubectl delete svc nginx

Test Numa

It can run emojivoto to test the system only runs on socket1, use it because it can general a lot of traffic between pods.

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml \
  | kubectl apply -f -

Now it can use htop to show all CPUs, and we can see most traffic on CPU 1~10 and 81~160. There are a little CPU usage on CPU 78 and 80, but it should be Linux regular scheduling(I have not add isocpu to host CPU this time)

The post ARM64 Kubernetes Qemu Metallb Linkerd Numa appeared first on richliu's blog.

Above is a figure to describe relationship about metallb/traefik and kubernetes.

Please reference previous article (Install Kubernetes on ARM64 Ubuntu 18.04)about how to setup environment.

Install Helm

First, install Helm, helm is a kubernetes packages manager. it can help to install traefik, for some package, like traefik, it’s not easy to install manually, I tried, but fail. Use helm to install is a good idea.

Following instruction can download and install helm, or you can visit helm webpage for latest version

Following is command to download and install helm.

wget https://get.helm.sh/helm-v2.15.1-linux-arm64.tar.gz
tar xvf helm-v2.15.1-linux-arm64.tar.gz
sudo mv linux-arm64/helm /usr/bin

Default helm(tiller seems doesn’t support ARM64 archeticture, so, it needs to use third party binary code for tiller. In here we used tiller-multiarch.

helm init --tiller-image=jessestuart/tiller 

Helm install error message

When you run helm install command, if see error message like below, please try to follow this step to solve it. (ref: TILLER AND ROLE-BASED ACCESS CONTROL and forbidden: User “system:serviceaccount:kube-system:default” cannot get namespaces in the namespace “default)

Create a rbac-config.yaml like following text
(ref. User “system:serviceaccount:kube-system:default” cannot get namespaces in the namespace “default” )

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: default

Run following command to create and upgrade helm initial

kubectl create -f rbac-config.yaml
helm init --service-account tiller --history-max 200 --tiller-image=jessestuart/tiller --upgrade

or run those commands directly.

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
helm init --upgrade --service-account tiller 

Install metallb

Automatically Install

It can use helm to install automatically, first create a metallb-config.yaml, contain like below.

configInline:
   address-pools:
   - name: default 
     protocol: layer2
     addresses:
         - 192.168.110.80-192.168.110.100 

Save it, and run follow command to apply it.

helm install --name=metallb --namespace=metallb-system -f  metallb-config.yaml stable/metallb 

Manually Install

It also can install metallb manually, run following command to install it.
It will install v0.8.1, it can visit metallb website to check latest version.

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml 

Add a new file name metallb.yaml and put follow text

apiVersion: v1
kind: ConfigMap
metadata:
   namespace: metallb-system
   name: config
data:
   config: |
     address-pools:
     - name: my-ip-space
       protocol: layer2
       addresses:
       - 192.168.110.70/28

Apply it

kubectl apply -f  metallb.yaml 

Install Traefik

Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience.

Following is command to initial helm and used helm to initial traefik.

helm install stable/traefik --name traefik --set dashboard.enabled=true,serviceType=NodePort,dashboard.domain=dashboard-traefik.techarea.org,rbac.enabled=true  --namespace kube-system

Traefik Dashboard

When used helm to install traefik, it also install traefik dashboard on dashboard-traefik.techarea.org.

Used following command to get traefik web service ip

kubectl describe svc traefik --namespace kube-system

It can see a http hosted on 10.44.0.2 , add 10.44.0.2 daskboard-traefik.techarea.org to browser PC’s hosts file, it can use browser to access traefik dashboard .

Create a nginx web site

In here, we create three nginx website, and assigned a external ip address, it will also assign a domain name, traefik will have load balance traffic to all three pods.

First create a new file called nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 4
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
  name: nginx
spec:
  rules:
  - host: nginx.techarea.org
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80

replicas means how many pods you went to create, in here, we used 3, it will create 3 pods on clients.
host means url, in here is nginx.techarea.org.

used kubectl to apply it.

kubectl apply -f nginx.yaml 

Get current service status

We can see external ip is 192.168.110.80, add 192.168.110.80 nginx.techarea.org to your browser PC’s host file, and used browser to access nginx.techarea.org, it can see welcome nginx.

Test Load Balance

Now we finished install traefik and metallb successfully, next step is to verified does that really work ?

it can use another host to run ab (apache benchmark to do this test) and used htop observe CPU loading.

Following is example to test nginx, if don’t see all CPU has loading, try to increase pods instance, like 4 .

ab -n 1000000 -c 400 -k http://nginx.techarea.org/

Below is example result for nginx load balance. 

See, that’s easy job, right ?

The post ARM64 Kubernetes Nginx Traefik Ingress Load Balance appeared first on richliu's blog.