這次升級要從 x64 平台升級到 ARM64 平台，雖然目前 Linux 下 ARM64 支援度還是不錯，但是兩個平台能用的東西還是有差異，主要是紀錄一下升級過程中有什麼需要注意的事情

ARM64 Hardware Prepareation

CPU Board: Ampere Altra CPU + AsRock ALTRAD8U : 這個組合現在已經能在 newegg 上買到了，也是目前最強大的 ARM64 平台，當 homelab 平台非常適合

DRAM: 買這種主機板用的都是 RDIMM ，有可能便宜有可能貴，可以查 Gigabyte 的 QVL List[Ref] 內的 DRAM list 購買，有朋友就是在日本買到DDR4 2933 32GB RDIMM 一隻5000羊的DDR，目前匯率 NT$1,100.- 非常便宜

Network: AsRock ALTRAD8U 有 1*Intel i210 Ethernet 和 2*10G Intel x550 Ethernet ，這部份到不是問題，我另外拿了舊的 Intel x540 4 Port ethernet 當小 switch 用

SATA: 這種板子預設都沒有 SATA interface 了，所以要買 SATA 卡，有 PCIe 轉 SATA ，這個要注意 PCIe bandwidth 和 SATA 數量，一分錢一份貨。
如果要追求速度，淘寶上也可以買到便宜 SAS 外接 SATA ，淘寶的 LSI SAS 8port internal 卡大部份 NT$2500 到 NT$6000 都有，記得買轉接線要看好規格，SAS 接頭的規格很多，一不小心就會弄錯

SSD: 板子上介面有 2* M.2 2280，如果覺得不夠，可以買 OCU 轉 U.2 轉接線，或是還有 PCIe slot ，可以買 PCIe 轉 M.2 都可以。但是 U.2 的硬碟不一定好買，而且常常價位偏高，但是相對上是比較便宜的選擇

Software Move To ARM64 Platform

QEMU/Libvirt

這部份應該是最難的部份，不過理論上也不是不可以，我原來是 VirtualBox ，但是 ARM64 沒有 VirtualBox ，只有 Proxmox 有，剛看了一下，還支援 riscv64,loongarch64 [ref]，不過我個人不太想再學一套，所以還是用 virt-manager/libvirtd 這套，一般用途 virt-manager 和 libvirtd 應該也很夠用了。

目前用到現在都可以用，Ubuntu 22.04，Windows 10 for x86，Windows 11 for ARM，安裝沒什麼問題。只有 Windows 10 for x86 非常慢，慢到我不想再開了，不知道為什麼。

Qemu bridge devices: 這個應該是問題，如果是以前用 VirtualBox ，可以直接選 NIC card interface ，但是 virt-manager 似乎沒這個選項，所以要直接設定到 bridge device，不過影響不是很大，就只是設定一個 bridge device，然後將 host internet ethernet port device 綁上去就好了

其他可能是如果 Libvirt 要用 graphic 輸出，不能用 qxl ，只能用 VGA or ramfb ，尤其是 windows 只能用 ramfb，如果 Guest 是 Linux 可以用 VGA ，顯示比較正常，如果連 GUI 都沒有，就直接看 console 輸出吧，問題更少

Qemu file sharing

QEMU 支援 virtiofs ，不過這個功能在 gentoo 內 ARM64 是不能用的，因為不能用，所以我也不知道 x64 那邊會有什麼限制
另外一個是 virtio-9p ，這個缺點是分享進 VM 之後，沒辦法使用 user 權限存取，用 root 身份限制也很多，像是存取 non-root user 檔案會有問題[ref]

有其他需求的人可以考慮改用 samba or nfs ，但是效能就會大大下降了，這點可能是有效能需求的人要考量的

Firewall

原來的 Firewall script 似乎沒辦法和 Libvirt 一起工作，我又不想修改，最後比較簡單的作法就是在我的 script 執行完之後，再重啟 libvirtd ，這樣就正常了。
正常後別忘了執行 /etc/init.d/iptables save 儲存，這樣重開機之後才會正常執行 firewall rule.

Docker

試了一下似乎沒什麼問題，但是目前沒什麼要跑的 APP ，所以應該測不出什麼
不過之前 arm64 k8s docker [ref]都跑過了，應該踩雷的機會低很多

Other Software

在 Linux 下轉移到 ARM64 平台問題不大，大部份的 software 都是安裝之後 copy 過去
就算是在系統上有資料的 mysql 和 apache 都是這樣

這次我用了一個方式轉移 configuration files
1. 將要轉移的程式設定檔 commit 到 git server
2. 從新機器將 git server 拉回來 configuration files
3. diff 之後將新的設定套上去

因為我是 gentoo 玩家，gentoo 升級 packages 常常需 merge configuration files 了，所以這個轉移的過程不難，只是很耗費時間而已。不過因此絕大部份的設定檔都沒有問題，反而是少數系統本身自帶的設定檔反而怪怪的

KDE

之前我都遠端跑 vnc + kde ，不過 Gentoo 有個 BUG 就是 plasmashell 的 CPU usage 會很高
這次換 virt-manager 之後可以用 ssh remote control VM，發現這個方式還不錯，反正我也只有用到這個功能，KDE 等有需求的時候再開 vnc 就好了，平時是不需要開的

搬移步驟

1. 備份系統碟，以備不時之需
2. 重新規劃電腦的網路卡，如果只有一張就直接設成 bridge interface
3. software raid 的部份可以參考自己之前的文章 Linux Software Raid 系統轉移
4. UEFI 的部份可以參考自己之前的文章 Gentoo ARM64 UEFI Install Guide
5. 搬移 APPs
5.1 重要的 APP 才要搬資料，ex: Web, mysql
5.2 不重要的 APP 只要搬設定檔，ex: apt-cache-ng, squid 等等
5.3 如果是切換平台，要確定 VM 沒問題才能切

Known Issues

IPv6 Default Router

不知道為什麼，在 wan 的 bridge interface 會增加一個 default route fe80::1 ，這個設定會讓 IPv6 失效，後來解決方式是在 LAN 端加上一個 IP address fe80::1 才解決

efi boot

因為我系統碟是做 software raid ，每次編完 kernel 再 update grub 時，記得 efi boot 時有時會 fail ，試著將兩顆 nvme 都 mount 上來加上再安裝一次 efi partition 似乎才會正常，找了一下沒找到原因，不管了，反正能動就好

grub-mkconfig -o /boot/grub/grub.cfg
umount /boot/efi
mount /dev/nvme0n1p1 /boot/efi
grub-install --target=arm64-efi --efi-directory=/boot/efi --bootloader-id=Gentoo
umount /boot/efi
mount /dev/nvme1n1p1 /boot/efi
grub-install --target=arm64-efi --efi-directory=/boot/efi --bootloader-id=Gentoo
umount /boot/efi

ref.
None

The post [Gentoo] Upgrade From x64 to ARM64 appeared first on richliu's blog.

Chinese article version: 在 ARM64 QEMU System 上安裝 ARM64 Windows 11

Nowadays, ARM64 systems have become relatively mature, and recently, Windows 11 for ARM64 can also be installed on these systems. Naturally, there are people who want to run it on QEMU. However, most articles on this topic are not well-organized.

Here, let’s document the things what I discoverd and need to be prepared.

Preparations Hardware and Software

1) System: Ampere Altra CPU
Small ARM64 dev boards like RPI4 or RPI5 should also work, but installing on them might not be necessary, because it only has limitation CPU power and Memory. A more cost-effective solution is to use Apple Mx series CPUs [ref], as they are also ARM64, and many people purchase them for installing Linux.

2) ARM64 ISO Image
While there are numerous options available on internet, for a relatively legal acquisition of the ISO image, it is recommended to use Windows-11-On-Ampere
This step can be a bit complex, but the key is to use UUP Dump to fetch the Windows ISO and then use Rufus to burn it onto a USB Flash drive. During the burning process, Rufus can override some of the restrictions of Windows 11, such as TPM Support, which may not be fully supported on ARM64 CPUs.

3) Windows Virtio Driver

Download the Windows Virtio driver and place it into the USB Disk that was created in the previous step.

Here has download URL can download the Windows virtio driver : Proxmox Windows VirtIO Drivers, alternative download website:  download the latest stable or  download the most recent.
After downloading, extract the ISO contents and place them onto the USB Disk.

4) Convert USB Dump to Image
Linux users are likely familiar with this step.

dd if=/dev/sda of=/storage/Win11_ARM64_INSTALL.img

Why go through these steps? The primary reason is that within QEMU, it seems Windows cannot simultaneously configure two ISO devices. I encountered a freeze when attempting to install using the straightforward method. Hence, I went through this elaborate process to make it install smoothly.

5) virt-manager
Most examples available to others involve pure QEMU commands. However, this time, I wanted to avoid typing a bunch of commands and preferred a more user-friendly approach. Therefore, the examples provided here utilize virt-manager. Nonetheless, I believe the underlying logic is the same.

New Virtual Machine on QEMU

Virt Type: KVM
Architecutre: aarch64
Machine Type: virt

Next step, choice the ISO image we just created Win11_ARM64_INSTALL.img, Choose “the operaion system you are installing: Microsoft Windows 11”

For the CPU and memory, I used 16 cores and 16GB, which should be sufficient. A smaller combination might work, but I haven’t tested it because I’m too lazy to do so.

It’s recommended to set up the Disk Image manually, virt-manager tends to allocate the entire specified disk space right from the start, which may not be space-efficient. Windows 11, with its minimum space requirement of 52GB, can be allocated around 60GB or more based on your needs. Here is example command to pre-create qemu image.

qemu-img create -f qcow2 MyImage.qcow2 60G

Before executing, choose “Customize configuration before install.” There are certain adjustments to make. Remember to enable the XML editing feature within virt-manager beforehand. Go to the main program, select Edit -> Preferences -> Enable XML editing, and ensure it is checked.

In the XML section related to CPUs, locate the segment pertaining to ‘hyperv,’ and modify it with the following settings:

    <hyperv mode="custom">
      <relaxed state="off"/>
      <vapic state="off"/>
      <spinlocks state="off"/>
    </hyperv>

In the CPU section, change the quantity from ‘sockets’ to ‘cores.’ Below is an example for 16 cores:

Add those hardware setting into current configuration:
Input -> USB Keyboard
Input -> EvTouch USB Graphics Tablet
Graphics -> Spice server
Video -> Ramfb

For the Video option, it is essential to choose Ramfb; otherwise, the graphics might not display properly. QXL or Virtio should not be used. Virtio-gpu-pci hasn’t been tested, but some suggest it might work[ref].

Lastly, in Storage, add a USB CDROM device and set it up with the Win11_ARM64_INSTALL.img file you created earlier.

In the Boot Options section, enable the option for USB CDROM 1, and then you should be ready to boot.

If the configuration is correct, you should be able to boot to this screen successfully.

If following the above steps results in a hang during the installation process, you can try adjusting the boot options by changing the boot order. Try setting “disk” first and “USB CD ROM” second.

Install Windows

Under normal circumstances, Windows might not detect the disk. In such cases, select “Load driver” -> “25982_1000_ARM64_EN-US(C:)” -> “Virtio-driver” -> “viostor” -> “win11” -> “ARM64” (Note: The labels and directories here may vary based on different configurations).

In the final step, for the NIC driver, when prompted to connect to the network upon the initial entry into Windows 11, select “Install driver” -> “25982_1000_ARM64_EN-US(D:)” -> “Virtio-driver” -> “NetKVM” -> “win11” -> “ARM64”.

With this, you should have successfully completed the installation of Windows 11.

Happy Playing Windows 11 on ARM64 Qemu

ref.
virtio-win / virtio-win-pkg-scripts

The post Install ARM64 Windows 11 on ARM64 QEMU System appeared first on richliu's blog.

English please see : Install ARM64 Windows 11 on ARM64 QEMU System

現在 ARM64 的系統相對成熟了，最近 Windows 11 for ARM64 也可以在系統上安裝，自然也會有人想弄到 QEMU 上去，不過看來看去，大部份都沒有整理完畢的文章

這邊記錄一下需要準備的東西

準備

1) 系統：Ampere Altra CPU
RPI4 or RPI5 這種開發板應該也可以，因為這些板子只有有限的 CPU Power 和記憶體，就沒有安裝 qemu 的必要性，便宜一點的解決方案就是 Apple Mx 系列的 CPU [ref]，因為也是 ARM64 ，而且很多人也是買來安裝 Linux

2) ARM64 ISO Image
網路上應該很多，不過如果要相對合法取得 ISO image ，推薦 Windows-11-On-Ampere
這步驟有點複雜，不過重點是用 UUP Dump 抓回 Windows ISO ，然後用 Rufus 燒到 USB Flash ，在燒錄的時候 Rufus 可以解除一些 Windows 11 的限制，像是 TPM Support 等等這些在 ARM64 CPU 還不完全支援的設定

3) Windows Virtio Driver

下載 Windows virtio driver，然後放進去剛剛那一步做到的 USB Disk

Proxmox Windows VirtIO Drivers 這邊有下載點， download the latest stable or  download the most recent ，都可以下載到 VirtIO ISO，下載完將 ISO 解開放到 USB Disk

4) 將 USB Dump 成 Image
Linux user 應該很熟這個步驟

dd if=/dev/sda of=/storage/Win11_ARM64_INSTALL.img

為什麼要這樣做？最大的原因就是 QEMU 內 Windows 現在似乎沒辦法同時設定二個 ISO 設備，我之前這樣裝會卡住。所以就繞了一大串讓它可以裝，還可以吧

5) virt-manager
其他人能找到的範例都是 qemu command ，不過我這次實在是不想打一堆指令，只想點點點，所以這次的範例都是用 virt-manager ，不過我相信背後的思路都是一樣的

New Virtual Machine

Virt Type: KVM
Architecutre: aarch64
Machine Type: virt

下一步，ISO image 選剛剛做好的 Win11_ARM64_INSTALL.img，Choose the operaion system you are installing: 選 Microsoft Windows 11

CPU 和記憶體我用 16 cores 和 16GB，小一點應該也可以，不過我懶得試了

如果空間不夠，Disk Image 建議自己設定，virt-manager 設定會一開始就 allocate 設定的 disk space ，不省空間，Windows 11 最小需求空間是 52G ，可以設 60G 或是更大，看需求
這邊是範例命令，可以預先建好硬碟空間

qemu-img create -f qcow2 MyImage.qcow2 60G

在執行之前選擇 Customize configuration before install，有些東西要修改，記得要先開啟 virt-manager 內的 XML 修改功能，在主程式下的 Edit -> Perferences -> Enable XML editing ，請勾選

在 CPUs 的 XML 這一段，找到 hyperv 這一段，改成如下的設定

    <hyperv mode="custom">
      <relaxed state="off"/>
      <vapic state="off"/>
      <spinlocks state="off"/>
    </hyperv>

在 CPU 數量，要從 sockets 改成 Cores，以下就是 16 Cores 的範例

Add hardware ，加入以下設定
Input -> USB Keyboard
Input -> EvTouch USB Graphics Tablet
Graphics -> Spice server
Video -> Ramfb

Video 選項一定要用 Ramfb ，否則 Graphics 會出不來，qxl or virtio 都不能用，virtio-gpu-pci 沒試過，有人說可以用[ref]
最後是 Storage 增加 USB CDROM device，並且將剛剛的 Win11_ARM64_INSTALL.img 設定上去

在 Boot Options 這邊 enable USB CDROM 1的選項，然後就可以開機了

如果設定正確，那就可以正常開機到這個畫面

如果照著以上步驟卻在安裝過程卡住了，可以試著調整 boot option 切換一下開機的順序
先 disk 後 USB CD ROM .

Install Windows

正常狀況下，Windows 是找不到 disk 的，要選 load driver -> 25982_1000_ARM64_EN-US(C:) -> Virtio-driver -> viostor -> win11 -> ARM64 (註：這部份的標籤和目錄可能因不同的設定而異）

最後一段，NIC driver ，在初次進入 Windows 11 時，要求要連線到網路時，選 Install driver -> 25982_1000_ARM64_EN-US(D:) -> Virtio-driver -> NetKVM-> win11 -> ARM64

這樣就可以完成安裝 Windows 11 了

Happy Playing Windows 11 on ARM64 Qemu

ref.
virtio-win / virtio-win-pkg-scripts

The post 在 ARM64 QEMU System 上安裝 ARM64 Windows 11 appeared first on richliu's blog.

My hardware also different, if you don’t have such powerful machine, it can skip numa part, that’s only for high end ARM64 machine.
The hardware is Ampere Mt. Collins which has 2*Ampere Altra 80 Cores CPU and 8*32G Memory, my experiment is run different VM on different CPU, to seperate them and lock it on different numa node.

There are two CPUs, so, it can create 3 nodes, so, I have plan to allocate resource like that.
1. k8s master on socket 0: 10 CPUs, 16G memory
2. k8s node1 on socket 0 : 70 CPUs, 100G memory
3. k8s node2 on socket 1: 80 CPUs, 120G memory

Kubernetes version v1.28.2
Ubuntu 22.04

Notice: Before reading this document, this document is personal experiment, not Ampere official document, take your own risk.

Prepare VM from libvirt

The server has remote control system which problem sufficent resolution to use GUI on server, so, I install ubuntu 22.04 on host system, than use virt-manager to create k8s master node, for qemu image, I create the disk image alone, seems create image when run install will allocate all disk space, seems that’s a virt-manager bug.
And the host network is configured to be a bridge interface, in my case, that’s br0.

So, when uses virt-manager to create VM, it can bridge network interface to br0.

Configure the VM with Numa

It can use numastat to check currnet numa status, this system has two numa nodes. it can use numastat and numactl to get more numa information.

# numastat -n

Per-node numastat info (in MBs):
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
Numa_Hit               591239.17       791075.41      1382314.58
Numa_Miss                   0.00            0.00            0.00
Numa_Foreign                0.00            0.00            0.00
Interleave_Hit              8.63            7.13           15.77
Local_Node             519434.17       777974.99      1297409.16
Other_Node              71805.00        13100.42        84905.42

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 128377 MB
node 0 free: 80224 MB
node 1 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 1 size: 126426 MB
node 1 free: 93888 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

virt-manager doesn’t have advance command to assign CPU, probably modify xml can get that purpose, but it more converince to use the “virsh” cli command to adjust VM parameters. “virsh list” show current VMs, “virsh vcpupin list current VM CPU scheduling”, “virsh vcpupin domain-id vcpu cpulist” to modify the VCPU run on which one CPU.

For memory, it needs to use “virsh numatune” to lock the memory on different node.
“virsh numatune domain [–mode mode] [–nodeset nodeset] [[–config] [–live] | [–current]]”, –current can update current running VM machine.

$ virsh list
 Id   Name          State
-----------------------------
 9    k8s-master   running
 10   k8s-node1     running

$ virsh vcpupin k8s-master
 VCPU   CPU Affinity
----------------------
 0      0-159
 1      0-159
 2      0-159
 3      0-159
 4      0-159
 5      0-159
 6      0-159
 7      0-159
 8      0-159
 9      0-159

# this command can lock 9 k8s-master's VCPU 0 on physical CPU 0. 
$ virsh vcpupin 9 0 0 

# Show current VM's numatune, for example, following is unset status. 
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   :

$ virsh numatune 9 --nodeset 0

# the numa_nodeset was set to 0
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   : 0

The problem is, seems this script need to run everytime, alternativate solution is to modify XML, but Mt. collis has 160 cores, I won’t to modify the XML file. script is more converience on this situation, following is my sample script to adjust k8s-master, k8s-node1, and k8s-node2’s parameter, it can easy to modify it if you install other system or AltraMax CPU.

#!/bin/bash

configure_vm(){
        local VM_NAME="$1"
        local VM_VCPU="$2"
        local VM_CPU="$3"
        local VM_NUMA="$4"
        local VM_TYPE="$5" # VM_TYPE for MASTER or NODE0, NODE1 .. self define name

        id=$(virsh list | grep $VM_NAME | awk '{print $1}')
        state=$(virsh list | grep $VM_NAME | awk '{print $3}')

        if [ "$state" != "running" ]; then
          echo "$VM_TYPE is not running"
          exit 1
        else
          echo "configure $VM_TYPE"
        fi
        numa_nodeset=$(virsh numatune $id | grep numa_nodeset | awk '{print $3}')

        start_cpu=$(echo $VM_VCPU | cut -d'-' -f1)
        end_cpu=$(echo $VM_VCPU | cut -d'-' -f2)
        for ((vcpu = start_cpu; vcpu <= end_cpu; vcpu++)); do
          # echo "$id $vcpu $VM_CPU"
          virsh vcpupin $id $vcpu $VM_CPU > /dev/null
        done
        if [ -n "$numa_nodeset" ]; then
          echo "$VM_TYPE numa_nodeset is $numa_nodeset"
        else
          # echo "$id $VM_NUMA"
          virsh numatune $id --nodeset $VM_NUMA
        fi

}

configure_vm "k8s-master" "0-9" "0-9" "0" "Master"
configure_vm "k8s-node1" "0-69" "10-79" "0" "Node0"
configure_vm "k8s-node2" "0-79" "80-159" "1" "Node1"


====

# After run the script, it can run numastat <pid> to see the qemu runs numa status. ex:
$ sudo numastat 206601

Per-node process memory usage (in MBs) for PID 206601 (qemu-system-aar)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                        11.51            0.00           11.51
Stack                        0.04            0.00            0.04
Private                   5984.93            0.00         5984.94
----------------  --------------- --------------- ---------------
Total                     5996.48            0.00         5996.48

In here, on socket 0 , 0-9 and 0-69 is vcpu, so, all start from 0. After run this script, it can use “virsh vcpupin” and “virsh numatune” to get current setting. Maybe cpuset parameter in XML can more easy to get this function, but I have not try it.

Install K8s

Here is my script, just reference some later blog and my previous configurtion script, so, it can copy and paste to system and run it directly.

Install basic system and k8s

# setup timezone
sudo apt install ntpdate -y
sudo ntpdate watch.stdtime.gov.tw
sudo timedatectl set-timezone Asia/Taipei

# install and enable docker

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update
sudo apt install -y containerd.io


containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

echo "----------------"
echo "   swapoff"
echo "----------------"
sudo swapoff -a
sudo sed -i '/swap/ s/^\(.*\)$/#\1/g' /etc/fstab

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update

sudo apt-get -y install kubelet
sudo apt-get -y install kubeadm
sudo apt-get -y install kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# Update kernel parametes
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# update sysctl
sudo sysctl --system

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

echo "if this is work node, it can run kubeadm join now"

echo "alias k=kubectl" >> .bashrc

Setup Master Node

Run this script on master node.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# echo "enable master to run pods"
# kubectl taint nodes --all node-role.kubernetes.io/master-

It will shows some message below, then run it on k8s-node1 and k8s-node2

kubeadm join 192.168.110.230:6443 –token e14f7k.35c3jpny81ry70re \

        –discovery-token-ca-cert-hash sha256:6636a7e4ec3542f19152f308f0789c52287b743b60efa3c13e7c23ce1147c581

If forgot command above, it can create another token to let other k8s node join, ex:

kubeadm token create --print-join-command

Flannel : Network fabric for container

K8s need a network fabric to provide internal network topology, there are many choices, but we will install matellb later, some other network like calico might have potential issue, so, we choice flannel as network fabric.

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

After run this command, the k8s node status will change to Ready, if not install network fabric, the node status will be NotReady.

$ kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   26h   v1.28.2
k8s-node1    Ready    <none>          26h   v1.28.2
k8s-node2    Ready    <none>          26h   v1.28.2

Helm: Kubbernetes Applications Manager

Install helm to further install metallb purpose, on previous article, helm2 needs another application to connect to k8s, but helm3 doesn’t need it, more easy to use on k8s.

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o /tmp/get_helm.sh
chmod 700 /tmp/get_helm.sh
sudo /tmp/get_helm.sh

Metallb: Bere metal load-balancer for K8s

Install k8s on private side, it needs load-balancer to provide external IP to communication with internet or intranet. If you install Metallb, the “kubectl get svc”, the EXTERNAL-IP shows “Pending”, and k8s doesn’t response the arp on host, probably it doesn’t configure kube-proxy, it you have experience on previous metallb installtion, it might miss this part.

PATH=$PATH:/usr/local/bin/

helm repo add metallb https://metallb.github.io/metallb
helm --namespace metallb-system install --create-namespace metallb metallb/metallb -f metallb-values.yaml

# https://metallb.universe.tf/installation/
# see what changes would be made, returns nonzero returncode if different
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl diff -f - -n kube-system

# actually apply the changes, returns nonzero returncode on errors only
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl apply -f - -n kube-system

metallb-values.yaml is also different like previous setting, it needs to configure L2Advertisement on metallb, without it, your metallb won’t work. Follwing is metallb-values.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: primary-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.110.200-192.168.110.219

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - primary-pool

Traefik: TBD

Linkerd

Just show the install script here

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

echo "export PATH=$PATH:/home/$USER/.linkerd2/bin" >> ~/.bashrc

linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -

Other

it can insert isolinux to host linux kernel parameter to isolate the Linux scheduling to k8s nodes. (TBD)

isolcpus=10-159 nohz=on nohz_full=10-159

Test with Nginx

It can deploy nginx to test this system

kubectl create deploy nginx --image=nginx
kubectl expose deploy nginx --port=80 --target-port=80 --type=LoadBalancer
kubectl get svc nginx
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
nginx   LoadBalancer   10.99.211.255   192.168.110.200   80:31011/TCP   25h

Now, it can try to access http://192.168.110.200.
if want to Delete nginx deploy, run following command.

kubectl delete deploy nginx
kubectl delete svc nginx

Test Numa

It can run emojivoto to test the system only runs on socket1, use it because it can general a lot of traffic between pods.

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml \
  | kubectl apply -f -

Now it can use htop to show all CPUs, and we can see most traffic on CPU 1~10 and 81~160. There are a little CPU usage on CPU 78 and 80, but it should be Linux regular scheduling(I have not add isocpu to host CPU this time)

The post ARM64 Kubernetes Qemu Metallb Linkerd Numa appeared first on richliu's blog.

下

systemctl umask libvertd 

就可以 unmask 了，這指令我實在是太少用，都記不起來以前有沒有碰過，第二關忘了是什麼錯誤訊息，大致上好像是沒辦法 link 到 libvirtd-ro 之類的，索性手動啟動所有的 libvirtd-* service

$ sudo systemctl start virtlogd
$ sudo systemctl enable virtlogd
$ sudo systemctl enable virtlockd
$ sudo systemctl start virtlockd
$ sudo systemctl start libvirtd-admin.socket
$ sudo systemctl enable libvirtd-admin.socket
$ sudo systemctl start libvirtd-ro.socket
$ sudo systemctl enable libvirtd-ro.socket
$ sudo systemctl enable libvirtd.socket
$ sudo systemctl start libvirtd.socket

最後應該是出現了一個 AppArmor 的錯誤訊息，像是這個

Unable to complete install: 'internal error: cannot load AppArmor profile 'libvirt-df1a8e70-7a27-41e4-a7cc-ea065d3b3699''

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createvm.py", line 2089, in _do_async_install
    guest.installer_instance.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 542, in start_install
    domain = self._create_guest(
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 491, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib/python3/dist-packages/libvirt.py", line 4034, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: internal error: cannot load AppArmor profile 'libvirt-df1a8e70-7a27-41e4-a7cc-ea065d3b3699'

試過了很多方式，都沒有辦法繞過 AppArmor ，最後只好放大絕，不要讓 qemu 使用 AppArmor ，不安全就算了對吧，

$ sudo vim /etc/libvirt/qemu.conf

# Set
security_driver = "none"

然後找到　security_driver 這個參數，設定成 “none” ，再重新啟動 libvirtd 就可以了
真是莫名其妙的 BUG ，因為其他台機器沒有碰到類似的問題

ref.
libvirt: error : unable to set AppArmor profile

The post Unable to complete install: internal error: cannot load AppArmor profile libvirt appeared first on richliu's blog.

EDITOR=vim sudo -E systemctl edit docker

加上這一段

[Service] 
ExecStartPre=/sbin/iptables -A FORWARD -p all -i br0 -j ACCEPT 
ExecStopPost=/sbin/iptables -D FORWARD -p all -i br0 -j ACCEPT

再重啟你的 Docker 就好了

sudo systemctl stop docker 
sudo systemctl start docker 

要不然就簡單一點，將 FORWARD rule 從預設 DROP 變成 ACCEPT

[Service] 
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT 

ref
[SOLVED] Docker breaks existing bridge I use for KVM/QEMU

The post QEMU Network Problem When Run With Docker appeared first on richliu's blog.

Add mac parameter to qemu command

in qemu command line, add mac= parameter to netdev option , ex:

-device virtio-net-device,netdev=net2,mac=52:54:00:56:12:34 \

Modify netplan

edit file /etc/netplan/50-cloud-init.yaml, and add section match, ex

network:
     version: 2
     ethernets:
         eth0:
             dhcp4: true
             match:
                 macaddress: 52:54:00:56:12:34
             set-name: eth0

now, you can shutdown qemu system and restart .

* Notice: if used ubuntu cloud image, please assign mac address before init it.
suppose that cloud image might save mac address some where .

The post Ubuntu 18.04 Change Mac Address on QEMU via Netplan appeared first on richliu's blog.

Ubuntu

It can refer to those three articles to create a new cloud image from Ubuntu server image, remember to change ARM64 image to x86_64 image .

Change Ubuntu Cloud Image Size (if you choice ubuntu 16.04 as your image)
Change Ubuntu Cloud Image Password
Ubuntu Cloud Image Taiwan Mirror Site

Install Qemu and setup bridge devices

apt install -y ovmf qemu bridge-utils
mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf
echo "allow virbr0" >> /etc/qemu/bridge.conf

Write this script to a file and run it.

SERVERFILE=xenial-server-cloudimg-amd64-uefi1.img
VERSION=`uname -r`

sudo qemu-system-x86_64 -name vm1 \
        -machine pc-i440fx-xenial,usb=off \
        -cpu qemu64 -m 8192 \
        -smp 12,sockets=1,cores=12,threads=1 \
        -nographic -nodefaults \
        -bios /usr/share/ovmf/OVMF.fd \
        -virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 \
        -drive file=$SERVERFILE  \
        -net nic,macaddr=$macaddr \
        -net tap,ifname=tap0 \
        -serial telnet::9001,server,nowait > guest1_log.txt 2>&1 &

sleep 5

brctl addif br0 tap0

it can use command
qemu-system-x86_64 –machine help
and
qemu-system-x86_64 –cpu help
to get all machine and cpu options.

it can change cpu and machine type here.

network would be bridge on tap0

below is good option to share folder between host and VM, below command will share /home/work with vm.

option 
--virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 

it can put this line to /etc/fstab to mount host’s folder to somewhere, in this case, just put the same directory like host does.

host0   /home/work   9p      trans=virtio,version=9p2000.L,user,umask=000   0 0

CentOS

CentOS document was written on 2021, but Ubuntu document on 2019, there is some difference between it. but that’s ok, I believe both ways are works.

Compile QEMU

CentOS default application doesn’t support run x86_64 code on ARM64 server, it needs to compile manually.

Please download qemu and compile it.

# Install CentOS Packages
yum config-manager --set-enabled powertools
yum install -y xorg-x11-xauth python3 ninja-build
yum install -y glib2-devel libmount-devel
yum install -y git glib2-devel libfdt-devel pixman-devel zlib-devel

# Download QEMU Source Code
wget https://download.qemu.org/qemu-5.2.0.tar.xz
# Decompress it
tar xvf qemu-5.2.0.tar.xz
# 
cd qemu-5.2.0
./configure --target-list="x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user"
make -j `nproc`
make install 

Then, qemu will be installed on /usr/local/bin

Download UEFI Image

It needs UEFI Image for x86_64 image boot up. It needs OVMF’s tianocore (Open Source UEFI BIOS) to boot up. And we can download kraxel as prebuild images.

# Install Kraxel into CentOS repos
cd /etc/yum.repos.d/
wget https://www.kraxel.org/repos/firmware.repo
yum install edk2.git-aarch64 edk2.git-ovmf-x64
cd <your vm directory>
cp /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd .
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd .

Prepare Bridge Devices

It needs bridge devices to connect to internet, default qemu bridge device name is virbr0 , due to CentOS doesn’t support brctl, follow this document(3 Ways to Create a Network Bridge in RHEL/CentOS 8) to create a CentOS bridge devices. Below is my experiment script to create bridge device, this setting will keep in system, reboot will not be reset.

nmcli conn add type bridge con-name virbr0 ifname virbr0

killall dhclient
ip addr flush enP7p2s0
nmcli conn modify virbr0 ipv4.addresses '192.168.3.191/16'
nmcli conn modify br0 ipv4.gateway '192.168.1.1'
nmcli conn modify br0 ipv4.dns '8.8.8.8'
nmcli conn modify br0 ipv4.method manual
nmcli conn add type ethernet slave-type bridge con-name bridge-br0 ifname enP7p2s0 master virbr0
nmcli conn up virbr0
nmcli conn up bridge-br0

nmcli conn show  --active

# Allow QEMU to access virbr0
mkdir -p /etc/qemu
echo "allow virbr0" >> /etc/qemu/bridge.conf

Run Qemu

It can refer above to prepare cloud image, I prefer to use ubuntu cloud image, but have not try CentOS image.

Remember Ubuntu 18.04 image won’t auto increase image size, please follow this Change Ubuntu Cloud Image Size to modify image size

qemu-system-x86_64 -m 8192 -smp 16 -nographic \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE-pure-efi.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=OVMF_VARS-pure-efi.fd \

        -hdc bionic-server-cloudimg-amd64.img \
        -netdev bridge,br=virbr0,id=net0 \
        -device virtio-net,netdev=net0,mac=00:11:22:33:44:55 \
        -serial telnet::9001,server,nowait > vmlog.txt 2>> vmlog.txt &

x86_64 application

Sometimes we don’t need to run whole VM to run some application just for x86_64, it can run user mode application directly. I am a lazy guy, so just reuse cloud image as x86_64 Linux library.

mkdir -p /home/amd64
qemu-img convert  bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.raw
losetup /dev/loop0 bionic-server-cloudimg-amd64.raw
kpartx -a /dev/loop0
mount /dev/mapper/loop0p1 /home/amd64

Umount Image

umount /home/amd64
losetup -d /dev/loop0

Run x86_64 only application

qemu-x86_64 -L /home/amd64/ bootutil64e
qemu-x86_64 -L /home/amd64/ nvmupdate64e -l

Unsolved Problem

Cannot find way to update INTEL NIC’s NVM

Tried vfio and application mode, vfio cannot pass iommu to host, it will fail.

User space application will cause mmap fail.

If you know how to update Intel NIC’s NVM, welcome to leave a message for me.

QEMU –static problem

Add –static parameter will cause some problem, cannot solve it right now.

./configure –target-list=”x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user” –static
===

ERROR: sizeof(size_t) doesn’t match GLIB_SIZEOF_SIZE_T.

    You probably need to set PKG_CONFIG_LIBDIR

    to point to the right pkg-config files for your    build target

The post QEMU Run x86_64/AMD64 VM on ARM64/AARCH64 appeared first on richliu's blog.

$ crash /usr/lib/debug/boot/vmlinux-4.18.0-25-generic 201907111618/dump.201907111618

crash: cannot determine page size

or (after add pagesize parameter )

$ crash /usr/lib/debug/boot/vmlinux-4.18.0-25-generic 201907111618/dump.201907111618 -p 65536

please wait… (gathering module symbol data)
WARNING: cannot access vmalloc’d module memory
please wait… (gathering task table data)
WARNING: duplicate idle tasks?
please wait… (determining panic task)
crash: invalid kernel virtual address: ffff0000100d0000 type: “64-bit KVADDR”

If you have problem like this, please update your Ubuntu crash version.
In my opinion, Ubuntu 18.04’s kdump crash only for kernel 4.15, not for HWE kernel or above.

In here, suggest to use HWE kernel or later for more hardware compatibility

download latest crash and compile it

vim /etc/apt/source.list
# removed all "#" before # deb-src
apt update
apt-get build-dep linux-crashdump
git clone https://github.com/crash-utility/crash.git
cd crash
make

and it can assign absolutely path to run new version crash, ex:

/root/crash/crash /usr/lib/debug/boot/vmlinux-5.0.0-20-generic 201907171611/dump.201907171611

Prepare QEMU Image and Run It

Use ubuntu cloud image as a QEMU Image would be fast and easy, it can download Ubuntu 18.04 cloud image first

wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-arm64.img

It also can download from local mirror site, like Taiwan mirror server

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img

After finished download job, it needs to change cloud image default password for login.
please refer another article Change Ubuntu Cloud Image Password. to change it.

Run QEMU With Cloud Image

Before run QEMU command, it needs to install necessary packages

sudo apt install -y qemu-efi qemu bridge-utils

Write bridge devices into qemu

mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf
echo "allow virbr0" >> /etc/qemu/bridge.conf

Create UEFI Bios image and flush for QEMU

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Configure you major LAN as bridge device, it can modify setup below to fit current network environment

ORIGNIC=enP6p1s0
ip addr flush $ORIGNIC
brctl addbr br0
brctl addif br0 $ORIGNIC
ifconfig br0 up
ifconfig br0 192.168.1.100 netmask 255.255.255.0
route add default gw 192.168.1.1
echo nameserver 8.8.8.8 >> /etc/resolv.conf

Save text below into a shell file and run it.

  
SERVERFILE=bionic-server-cloudimg-arm64.img

 sudo qemu-system-aarch64 -name kdump-vm \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -smp 8,sockets=1,cores=8,threads=1 \
         -nographic -nodefaults \
         -drive if=pflash,format=raw,readonly,file=flash0.img \
         -drive if=pflash,format=raw,file=flash1.img \
         -drive file=$SERVERFILE,if=none,id=disk1  \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0 \
         -serial telnet::9001,server,nowait > vm_log.txt 2>&1 &
 sleep 5
 brctl addif br0 tap0 

it can use command “telnet localhost 9001” to login qemu image .
if failed, please check vm_log.txt for reason.

if there is blank screen on VM, please check whether copy right qemu-efi into flash0.img.

Running Kdump in VM

It can install openssh-server in VM, thus, it would be more easy to run command remotely. especially, if used telnet, vim usually cannot reflect right terminal window size and cannot wrap word correctly.

Install kdump (only for ARM64 QEMU, if used HEW kernel, it needs to download latest version linux-crashdump)

sudo apt install linux-crashdump

More information about install kdump, please refer to this article Kernel Crash Dump

I would suggest to increase kdump ramdisk size to 512M, not used ubuntu linux-crashdump default size. edit /etc/default/grub.d/kdump-tools.cfg and find your RAM size, change following parameter to 512M, ex: my VM memory size is 8GB, so, I modify 4G-32G paramete to 4G-32G:512M

GRUB_CMDLINE_LINUX_DEFAULT=”$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M”

After update kernel parameter, update grub to make sure it will get new parameter on boot

update-grub2

Install Kernel Debug Symbol

Install GPG Key

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622

Add repository config

codename=$(lsb_release -c | awk  '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list < < EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

Update packages and install symbol. (ref Installing Ubuntu Kernel Debugging Symbols)

sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

Trigger Kdump

it can used command

kdump-config show 

to check current kdump status.

$ kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0xdfe00000
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.15.0-54-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-54-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p –command-line=”BOOT_IMAGE=/boot/vmlinuz-4.15.0-54-generic root=UUID=6242429e-2876-4d52-a590-99654a6a91ac ro quiet splash vt.handoff=1 nr_cpus=1 systemd.unit=kdump-tools-dump.service” –initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

if evenything is ok, it should have state: ready to kdump .

Trigger Kdump

it can easy to trigger Kdump via sysrq. I will suggest to trigger sysrq on telnet. it will show dump message and reboot message. more easy to know it get right result.

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

First command to configure sysrq to accept all trigger.
Second command is to trigger kernel crash

$ echo c > /proc/sysrq-trigger
[ 31.855103] sysrq: SysRq : Trigger a crash
[ 31.861713] Internal error: Accessing user space memory outside uaccess.h routines: 96000044 [#1] SMP
[ 31.869136] Modules linked in: nls_iso8859_1 sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear aes_ce_blk aes_ce_cipher crc32_ce
crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_net virtio_blk aes_neon_bs aes_neon_blk crypto_simd cryptd aes_arm64
[ 31.879776] CPU: 1 PID: 1385 Comm: bash Not tainted 4.15.0-54-generic #58-Ubuntu
[ 31.882430] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 31.884892] pstate: 00400005 (nzcv daif +PAN -UAO)
[ 31.886641] pc : sysrq_handle_crash+0x24/0x30
[ 31.888198] lr : __handle_sysrq+0xbc/0x1c0
[ 31.889627] sp : ffff00000e513d50
[ 31.890835] x29: ffff00000e513d50 x28: ffff8001f2e93c00
[ 31.892723] x27: ffff000008b32000 x26: 0000000000000040
[ 31.894639] x25: 0000000000000124 x24: ffff0000095ee000
[ 31.896524] x23: 0000000000000004 x22: 0000000000000002
[ 31.898433] x21: 0000000000000063 x20: ffff000009550000
[ 31.900309] x19: ffff0000095ee4e0 x18: ffffffffffffffff
[ 31.902229] x17: 0000000000000000 x16: 0000000000000000
[ 31.904127] x15: ffff000009528c08 x14: ffff0000896d687f
[ 31.906046] x13: ffff0000096d688d x12: ffff00000954f000
[ 31.907954] x11: ffff000009529660 x10: ffff000008708540
[ 31.909848] x9 : 00000000ffffffd0 x8 : 0000000000000017
[ 31.911801] x7 : 6767697254203a20 x6 : ffff8001ffdc82e8
[ 31.913655] x5 : ffff8001ffdc82e8 x4 : 0000000000000000
[ 31.915569] x3 : ffff8001ffdd06c8 x2 : 068be67f3165eb00
[ 31.917430] x1 : 0000000000000000 x0 : 0000000000000001
[ 31.919312] Process bash (pid: 1385, stack limit = 0x (ptrval))
[ 31.921599] Call trace:
[ 31.922506] sysrq_handle_crash+0x24/0x30
[ 31.923964] __handle_sysrq+0xbc/0x1c0
[ 31.925311] write_sysrq_trigger+0xd8/0x120
[ 31.926851] proc_reg_write+0x80/0xc0
[ 31.928178] __vfs_write+0x48/0x80
[ 31.929399] vfs_write+0xac/0x1b0
[ 31.930618] SyS_write+0x6c/0xd8
[ 31.931777] el0_svc_naked+0x30/0x34
[ 31.933062] Code: 52800020 b90ca020 d5033e9f d2800001 (39000020)
[ 31.935269] SMP: stopping secondary CPUs
[ 31.937745] Starting crashdump kernel…
[ 31.939108] Bye!

After kdump save current status and reboot , it will save crash log on /var/crash, run crash command below, it can see some output

$ crash –mod /usr/lib/debug/lib/modules/4.15.0-54-generic/ /usr/lib/debug/boot/vmlinux-4.15.0-
54-generic 201907150556/dump.201907150556
crash 7.2.1
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter “help copying” to see the conditions.
This program has absolutely no warranty. Enter “help warranty” for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “aarch64-unknown-linux-gnu”…
KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-54-generic DUMPFILE: 201907150556/dump.201907150556 [PARTIAL DUMP] CPUS: 12 DATE: Mon Jul 15 05:56:07 2019 UPTIME: 2135039823346 days, 00:15:35
LOAD AVERAGE: 0.41, 0.32, 0.13
TASKS: 202
NODENAME: ubuntu
RELEASE: 4.15.0-54-generic
VERSION: #58-Ubuntu SMP Mon Jun 24 10:56:40 UTC 2019
MACHINE: aarch64 (unknown Mhz)
MEMORY: 8 GB
PANIC: “sysrq: SysRq : Trigger a crash”
PID: 1444
COMMAND: “bash”
TASK: ffff8001f019ad00 [THREAD_INFO: ffff8001f019ad00]
CPU: 1
STATE: TASK_RUNNING (SYSRQ)

Next step, more how to use crash information, please refer to article Analyzing Linux kernel crash dumps with crash – The one tutorial that has it all

The post Kdump in ubuntu 18.04 ARM64 appeared first on richliu's blog.

modify /etc/libvirt/qemu/networks/default.xml can change that

The post QEMU change default nat address configure file appeared first on richliu's blog.

Add this line to QEMU command

-virtfs local,path=/path/to/share,mount_tag=host0,security_model=passthrough,id=host0

and in guest os, edit /etc/fstab and add

host0   /wherever    9p      trans=virtio,version=9p2000.L   0 0

ref.
https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu

The post QEMU Share Directory with Host Without Networking appeared first on richliu's blog.

安裝

$ sudo apt install virt-manager

跑一個簡單的 aarch64 image

$ sudo virt-install --name ubuntu-aarch64 --ram 2048 --arch aarch64 --boot uefi \
--disk /home/test/ubuntu.img,device=disk,bus=virtio \
--boot machine=virt,kernel=trusty-kernel,kernel_args="console=ttyAMA0 root=/dev/vda1 rw kgdboc=ttyAMA0,115200" \
--network bridge:virbr0 \
--os-type=linux \
--nographics \
--noautoconsole

ubuntu.img 是預先準備好的 ubuntu cloud image, 16.04 才可以支援比較多的 ARM64 平台.
14.04 支援的就少很多.

以上的參數比較重要的是 –noautoconsole, 這樣執行時就不會等在那邊, 會自動啟動.

–noautoconsole will install vm to libvirt without connect to console

—- 2017/03/10 —-
也可以用 virt-install + netboot install

先建立 image

$ qemu-img create -f qcow2 ./name.qcow2 30G 

再執行 virt-install , 目前測過 14.04 和 16.04 , 只有 16.04 成功, 猜想是 ports 那邊少了什麼東西

$ sudo virt-install --name ubuntu-aarch64 --ram 4096 --vcpus 4 --arch aarch64 \
        --boot uefi \
        --disk /home/richliu/disk.qcow2,device=disk,bus=virtio \
        --network bridge:virbr0 \
        --os-type=linux \
        --nographics \
        --location 'http://ports.ubuntu.com/dists/xenial/main/installer-arm64/'

這個參數可以順利安裝完, 然後再啟動系統就好

—- 2017/04/26 —-
如果是要裝 ARM64 Trusty-Update 的 Netboot kernel . 要動一下小技巧
1) 先在 host VM 上安裝 apache2
2) 到 /var/www/html 上 mirror http://ports.ubuntu.com/dists/trusty-updates/main/installer-arm64/current/images/ 下的
* MANIFEST
* MANIFEST.udebs
* xenial-generic/netboot =>

目錄要照著放, 所以會是

$ cd /var/www/html
$ mkdir -p dists/trusty-updates/main/installer-arm64/current/images/
$ cd dists/trusty-updates/main/installer-arm64/current/images/
$ lftp http://ports.ubuntu.com/dists/trusty-updates/main/installer-arm64/current/images/
[download blah blah]
$ mv xenial-generic/netboot .

然後 –location 指到 http://[ARM64 Server IP] 這樣就可以了, ex:

virt-install --name ubuntu-aarch64 --ram 8192 --vcpus 4 --arch aarch64 \
--boot uefi \
--disk /root/disk.qcow2,device=disk,bus=virtio \
--network bridge:virbr0,model=virtio \
--os-type=linux \
--nographics \
--location http://10.102.81.16/dists/trusty-updates/main/installer-arm64

對了, 這邊的 model=virtio 是強迫 network interface 不要用 vhost, 而是用 virtio

$ virsh start ubuntu-aarch64
$ virsh connect ubuntu-aarch64

ref.
KVM Guest virt-install examples with virtio drivers
Installing Virtual Machines with virt-install, plus copy pastable distro install one-liners

The post aarch64/ARM64 virt-install 啟動方法 appeared first on richliu's blog.