In previous article on How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform , we installed the Nvidia driver and ran PyTorch on the platform. Now, this platform has Nvidia RTX gaming card, can we use it to play game?

The answer is yes, there are some windows x86 or x64 emulators on Linux, but the most stable solution comes from Ampere’s official release solution. Other emulators might not work as well on this combination, such as FEX, but I don’t know whether it is a driver issue or another issue.

Basically, following Ampere’s documentation makes it easy to install the driver and play games. Let’s combine both documents into one and simplify it in this article.

Hardware
CPU: Ampere Altra
Board: AsRock ALTRAD8UD
GPU: Nvidia RTX 4080

Reference Ampere’s document
NVIDIA GPU Accelerated Linux Desktop on Ampere
Steam-on-Ampere

Install Nvidia Driver

All the conditions are similar to those outlined in the article How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform, using Ubuntu 22.04 with a general kernel (not the HWE kernel, which Nvidia may address in the future).

I haven’t utilized the original document’s script; instead, I’ve copied some necessary commands from it.

sudo apt-get install build-essential
# Install Nvidia driver here
sudo rmmod nouveau
cat << EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
EOF

sudo nvidia-xconfig -a --cool-bits=31 --allow-empty-initial-configuration

# Install Desktop, personally I like kde. 
sudo apt install kde-plasma-desktop
# or it can install regular ubuntu desktop 
# sudo apt install ubuntu-desktop

Download Nvidia ARM64 (aarch64) Display Driver ; the current version, 535.154.05, is acceptable. After installing the Nvidia driver, simply reboot the system. You can then use the following command to check the driver status.

nvidia-smi

Disable aspm from kernel

I didn’t apply this option. Most of the time, it’s only necessary to do so if the system encounters the massive AER error issue. Here’s an example command from the original document:

sudo nano /etc/default/grub
# Add pcie_aspm=off to kernel parameters
# GRUB_CMDLINE_LINUX_DEFAULT="pcie_aspm=off"
sudo update-grub

Install Box86

Following commands show how to install Box86

sudo wget https://itai-nelken.github.io/weekly-box86-debs/debian/box86.list -O /etc/apt/sources.list.d/box86.list
wget -qO- https://itai-nelken.github.io/weekly-box86-debs/debian/KEY.gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/box86-debs-archive-keyring.gpg
sudo dpkg --add-architecture armhf
sudo apt update
sudo apt install box86:armhf -y

Install Box64

Following command show how to install Box64, but original instruction have update , a package name was changed.

sudo wget https://ryanfortner.github.io/box64-debs/box64.list -O /etc/apt/sources.list.d/box64.list
wget -qO- https://ryanfortner.github.io/box64-debs/KEY.gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/box64-debs-archive-keyring.gpg
sudo apt update 
sudo apt install box64-arm64 -y
sudo systemctl restart systemd-binfmt

Installing Steam

Because I am using the Ubuntu 22.04 ARM64 server version with minimal, so, some packages might not be installed, Before Install Steam, install necessary packages

sudo apt install lsof

sudo apt install zenity
sudo apt install alsa

Use the following to install steam.

git clone https://github.com/ptitSeb/box86
cd box86
./install_steam.sh

Now, it can use steam now.

/usr/local/bin/steam

Okay, log in and enjoy it. However, in my experience, when installing other emulators, it might damage the box86+Steam setup, necessitating a system reinstallation to fix it.

Furthermore, the game’s performance seems not as good as expected, possibly due to my choice of Vampire Survivors and enabling high performance

Xbox JoyStick controller

I have an Xbox controller, and ALTRAD8UD doesn’t have Bluetooth capability, so I cannot use wireless conntions. However, it can connect via USB.

There are several Xbox drivers available for Linux. After testing, I found that xpadano is compatible with this platform. Simply follow their instruction guide to install the Xbox driver.

# Install DKMS package
sudo apt-get install dkms linux-headers-`uname -r`

# download source code
git clone https://github.com/atar-axis/xpadneo.git
cd xpadneo
# use DKMS to compile the driver
sudo ./install.sh

The Xbox controller driver will be installed automatically, and it works well in my tests. You can install the jstest package to perform a joystick test.

The Xbox controller functions seamlessly on Steam and Vampire Survivors. I can easily use the Xbox controller to play Vampire Survivors.

The post Play Game On Ampere Altra ARM64 Platform appeared first on richliu's blog.

• * The Ubuntu/Fedora rootfs on tftp/nfs server:
  It can refer to the my article MIPS/ARM64 Debian/Ubuntu debootstrap for building the Ubuntu rootfs.
  If want to build the Fedora rootfs, it can install a new system on qemu and copy it from the qcow2 image.

• * PXE Server:
  You can refer my article ARM64 Centos8/Ubuntu PXE Boot Install on Ubuntu Server to build a PXE server.

• NFS Server:
  Assume the rootfs for Ubuntu is on /nfs/ubuntu2204 and Fedora 36 is on /nfs/fedora36.

Ubuntu PXE Boot

Copy vmlinux and initramfs to tftp server. Copy your current system’s vmlinux to your tftpboot. For example, create a directory for Ubuntu 22.04 in /tftpboot/nfs/ubuntu2204, and copy the vmlinuz to the folder using the following command:

$ cp /boot/vmlinuz-$(uname -r) /tftpboot/nfs/ubuntu2204/vmlinuz

Copy the initrd.img to the folder using the following command:

$ cp /boot/initrd.img-$(uname -r) /tftpboot/nfs/ubuntu2204/initrd.img

Setup your PXE server’s grub.cfg and add a record. Use the following example for the record:

menuentry 'Ubuntu22.04 nfsboot' { 
    linux nfs/ubuntu2204/vmlinuz ip=dhcp root=/dev/nfs nfsroot=<NFS server IP>:/nfs/ubuntu2204,nolock rw netboot=nfs 
    initrd nfs/ubuntu2204/initrd.img 
}

Fedora PXE Boot

The procedure is similar to Ubuntu, but the default Fedora initramfs doesn’t support nfs boot.
1. Rebuild the initramfs with dracut. You need to add nfs and network support to your Fedora initramfs using the following command:

$ dracut --add nfs --add network --force

It will generate the same name vmlinuz and initramfs. Copy it to your directory just like Ubuntu.
2. Disable selinux.
Default Fedora selinux is enabled, but it cannot enable on nfs rootfs (or it can but needs to set up something). You can disable it by adding selinux=0 on boot kernel parameter. Use the following example for the record:

linux nfs/fedora36/vmlinuz boot=nfs root=/dev/nfs nfsroot=<NFS server IP>:/nfs/fedora36,nolock rw netboot=nfs ip=dhcp selitarnux=0

The post Ubuntu/Fedora PXE Boot appeared first on richliu's blog.

Hardware topology 

[ARM64 CLIENT] <——-> [Server]  <——-> Internet 

SERVER ethernet ip is 10.0.0.1 , another interface connect to internet. 

Install necessary packages 

Install TFTP, HTTP and DHCPD Server and rpm2cpio 

$ apt install tftpd-hpa apache2 isc-dhcp-server rpm2cpio

Modify dhcpd server configuration 

$ vim /etc/dhcp/dhcpd.conf

following is example configuration,it can be changed to fit your environment 

authoritative;
default-lease-time                      259200;
max-lease-time                          518400;
option routers                          10.0.0.1;
option subnet-mask                      255.255.255.0;
option domain-name-servers              168.95.1.1,8.8.8.8;
ddns-update-style                       interim;
ignore client-updates;
allow booting;
allow bootp;
allow unknown-clients;
server-name                             pxis;

subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.100 10.0.0.200;
    option ip-forwarding off;
    option subnet-mask                  255.255.255.0;
    next-server 10.0.0.1;
    filename "shim.efi";

}

Modify tftpd setting 

$ vim /etc/default/tftpd-hpa

Add –create to TFTP_OPTIONS and change tftp directory to /tftpboot 

following is example configuration file 

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/tftpboot"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure  --create"

Restart Service 

mkdir /tftpboot 
/etc/init.d/tftpd-hpa restart
/etc/init.d/apache2 restart
/etc/init.d/isc-dhcp-server restart

Get ARM64 pxeboot file 

some times the shim and grub2-efi would be erase, just visit mirror.centos.org and find new one than download it.

cd /tftpboot 
wget http://mirror.centos.org/centos/8-stream/BaseOS/aarch64/os/Packages/shim-aa64-15-15.el8_2.aarch64.rpm
wget http://mirror.centos.org/centos/8-stream/BaseOS/aarch64/os/Packages/grub2-efi-aa64-2.02-106.el8.aarch64.rpm
rpm2cpio shim-aa64-15-15.el8_2.aarch64.rpm | cpio -dimv
rpm2cpio grub2-efi-aa64-2.02-106.el8.aarch64.rpm | cpio -dimv
cp ./boot/efi/EFI/centos/shim.efi .
cp ./boot/efi/EFI/centos/grubaa64.efi .
chmod +rx *.efi

Configure grub.conf 

$ vim /tftpboot/grub.cfg

Following is grub.conf example 

set timeout=60
menuentry 'CentOS' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}

If your system support secure boot and you want to do it, please replace linux with linuxefi and replace initrd with initrdefi. 

Get CentOS 8 stream image and copy it to web server 

cd /var/www/html 
wget http://ftp.twaren.net/Linux/CentOS/8-stream/isos/x86_64/CentOS-Stream-8-x86_64-20220215-dvd1.iso
mkdir disk 
mount -o loop CentOS-Stream-8-x86_64-20220215-dvd1.iso disk
cp -a disk centos8 
umount disk 
rmdir disk

# copy boot file into tftp directory 

cd /tftpboot
cp /var/www/html/centos8/images/ . -a

After copy all file into http file directory, it can visit via browser on server with URL http://10.0.0.1/centos8/ to check whether file exist or not. 

NAT

Server should support NAT function to forward packet. 

assume interface connect to internet is wls3
Following is a sample command to enable Linux NAT function which can make client to access internet or intranet . 

$ echo 1 > /proc/sys/net/ipv4/ip_forward ==> enable it immediately 
$ iptables -t nat -A POSTROUTING -o wls3 -j MASQUERADE

Now, you can choose pxeboot on your ARM64 machine. if everything successful, you can see CentOS on your console if everything is ok . 

Known Issue

Stdin: Invalid argument

My system cannot boot into the Ubuntu install program for an unknown reason. After checking the system, it seems that the issue may be caused by an inability to access internet resources for some reason, even though the DNS and network appear to be fine. An alternative solution is to put the ISO on a website and assign the argument to the parameter. Here is an example using Ubuntu 22.04.

menuentry 'Ubuntu22.04 Net Install' {
linux ubuntu2204/vmlinuz ip=dhcp url=http://10.0.0.1/ubuntu-22.04.1-live-server-arm64.iso only-ubiquity
initrd ubuntu2204/initrd
}

Debug 

Wireshark is your good friend, use it. 

Ubuntu PXE Environment

It can download Ubuntu Netboot images from this url.
http://cdimage.ubuntu.com/netboot/

Choice what you want, for example, url below is Ubuntu 18.04 for ARM64 image
http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/
Ubuntu 20.04 Netboot file
http://ports.ubuntu.com/ubuntu-ports/dists/focal-updates/main/installer-arm64/current/legacy-images/netboot/netboot.tar.gz

Download netboot image to tftp directory, I prefer to put different version to different directory.

$ cd /tftpboot
$ wget http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/netboot.tar.gz
$ mkdir ubuntu1804
$ cd ubuntu1804
$ tar xvf ../netboot.tar.gz

Modify grub.cfg and add Ubuntu option to grub.cfg, save it, now, this server can also provide ubuntu net install

set timeout=60
menuentry 'CentOS8 Install' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}
menuentry 'Ubuntu18.04 Install' {
  linux ubuntu1804/ubuntu-installer/arm64/linux ip=dhcp
  initrd ubuntu1804/ubuntu-installer/arm64/initrd.gz
}

Ubuntu ISO environment

It can download iso file and use the iso file as install media.

in Linux, download Ubuntu 20.04 ARM64 iso file

$ wget https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.3-live-server-arm64.iso
$ mount -o loop ubuntu-20.04.3-live-server-arm64.iso disk
$ mkdir ubuntu2004
$ cp disk/casper ubuntu2004/ -a 
$ umount disk

Modify grub.cfg as below (not test)

set timeout=60
menuentry 'CentOS8 Install' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}
menuentry 'Ubuntu20.04 Install' {
  linux ubuntu2004/casper/vmlinuz ip=dhcp
  initrd ubuntu2004/casper/initrd
}

It also can use HWE kernel

menuentry 'Ubuntu20.04 Install' {
  linux ubuntu2004/casper/hwe-vmlinux ip=dhcp
  initrd ubuntu2004/casper/hwe-initrd
}

Note for GRUB Install Command

CentOS

svr=10.0.0.1; root=boot/centos8/images/pxeboot
linux (http,$svr)/$root/vmlinuz inst.stage2=http://$svr/boot/centos8 text
initrd (http,$svr)/$root/initrd.img

Ubuntu

svr=192.168.110.254; root=boot/ubuntu2004/casper
iso=ubuntu-20.04.3-live-server-arm64.iso
linux (http,$svr)/$root/vmlinuz url=http://192.168.110.254/boot/$iso only-ubiquity ip=dhcp ---
initrd (http,$svr)/$root/initrd

linux (http,$svr)/$root/hwe-vmlinuz url=http://192.168.110.254/boot/$iso only-ubiquity ip=dhcp ---
initrd (http,$svr)/$root/hwe-initrd

Fedora 36

Fedora’s setting is more closed to CentOS, due to CentOS might be dropped recently, so, moved to Fedora environment is necessary.

Get initrd/vmlinux image

Other parts like CentOS,
Download the PXE initrd and vmlinuz image, free.nchc.org.tw is Taiwan local Fedora server, you can replace it with local server.

mkdir -p /tftpboot/f36

cd /tftpboot/f36
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/os/images/pxeboot/initrd.img
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/os/images/pxeboot/vmlinuz

cd /var/www/html
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/iso/Fedora-Server-dvd-aarch64-36-1.5.iso
mkdir disk 
mount -o loop Fedora-Server-dvd-aarch64-36-1.5.iso disk
cp -a disk f36
umount disk

Grub Fedora Setting

Add following grub entry text into /tftpboot/grub.cfg, thus, it can insert a “Fedora36” menu on pxe boot.

menuentry 'Fedora36' {
  linux f36/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/f36/
  initrd f36/initrd.img
}

Ref. 

https://docs.centos.org/en-US/centos/install-guide/pxe-server/#sect-network-boot-setup-uefi
https://docs.centos.org/en-US/8-docs/advanced-install/assembly_preparing-for-a-network-install/
https://forums.opensuse.org/showthread.php/501075-error-cannot-find-commands-linuxefi-or-initrdefi

The post ARM64 Centos8/Ubuntu PXE Boot Install on Ubuntu Server appeared first on richliu's blog.

Environment

Hardware : Ampere eMag Server with 128G RAM and 1T storage
Kubernetes master and nodes are running in QEMU image, so, we prepared three QEMU ARM64 ubuntu 18.04 images for kubernetes running. It also can use two VM to run this demo.

Prepare QEMU Image

It can download Ubuntu bionic cloud image from Taiwan mirror site

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img 

or international mirror site

https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-arm64.img

Install Necessary Package

Run following command to install necessary packages .

apt install -y qemu-system-arm bridge-utils cloud-image-utils git qemu-efi

Increase QEMU image size

qemu-img resize bionic-server-cloudimg-arm64.img +200G

Create UEFI bios and nvram for qemu image boot.

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Copy Image to three spare images

cp flash0.img flash0-1.img
cp flash0.img flash0-2.img 
cp flash0.img flash0-3.img 
cp flash1.img flash1-1.img
cp flash1.img flash1-2.img
cp flash1.img flash1-3.img
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-1.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-2.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-3.img 

Set up bridge and network devices

We already install bridge-utils before, now, it should configure bridge devices.

 mkdir -p /etc/qemu ; echo "allow br0" > /etc/qemu/bridge.conf 

Because there is only one network interface on machine, we should use this interface as bridge device, save text below (name nic-br.sh or what you like) and modify it to fit your environment.

ORIGNIC=enP6p1s0
ip addr flush $ORIGINIC
brctl addbr br0
brctl addif br0 $ORIGINIC
ifconfig br0 up
ifconfig br0 192.168.110.1 netmask 255.255.0.0
route add default gw 192.168.1.1
echo nameserver 8.8.8.8 >> /etc/resolv.conf 

Configure Hugepage On Host Site (Option)

Due to previous research on dpdk runs on ARM64 server, so, configure hugepages here. But not necessary for run k8s environment.
You can ignore this part

Configure Line Kernel boot command

It needs to reserve hugepages memory space for Linux kernel, it reserves 32 pages and each pages 1GB, total 32GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=32 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX="console=tty0 hugepagesz=1GB hugepages=32"

then run foloowing command to update grub.cfg

 update-grub 

Mount hugetlbfs by default

If wants to use hugepage, it needs to mount hugetlbfs somewhere, we can create a directory, ex: /mnt/hugepages and mount hugetblsfs there by default.

mkdir -p /mnt/hugepages 
echo "nodev   /mnt/hugepages                  hugetlbfs       defaults,pagesize=1G    0 0 none" >> /etc/fstab 

Now, please reboot to make huagpage works.

Running QEMU

Following is QEMU script, save it (ex: named run-qemu.sh). It can be more simplification, but this script would be more understand what different.

IMAGE1=bionic-server-cloudimg-arm64-1.img
sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-1111-1111-1111-11111aaaaaaa \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-1.img -pflash flash1-1.img \
         -drive file=$IMAGE1,if=none,id=disk1 \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0,mac=52:54:00:11:11:11 \
         -mem-path /mnt/hugepages \
         -serial telnet::9001,server,nowait > guest_log-1.txt 2>> guest_log-1.txt &
IMAGE2=bionic-server-cloudimg-arm64-2.img
sudo qemu-system-aarch64 -name vm2 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-2222-2222-2222-22222bbbbbbb \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-2.img -pflash flash1-2.img \
         -drive file=$IMAGE2,if=none,id=disk2 \
         -device virtio-blk-device,scsi=off,drive=disk2,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net2,ifname=tap1 \
         -device virtio-net-device,netdev=net2,mac=52:54:00:22:22:22 \
         -mem-path /mnt/hugepages \
         -serial telnet::9002,server,nowait > guest_log-2.txt 2>> guest_log-2.txt &
IMAGE3=bionic-server-cloudimg-arm64-3.img
sudo qemu-system-aarch64 -name vm3 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-3333-3333-3333-33333ccccccc \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-3.img -pflash flash1-3.img \
         -drive file=$IMAGE3,if=none,id=disk3 \
         -device virtio-blk-device,scsi=off,drive=disk3,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net3,ifname=tap2 \
         -device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \
         -mem-path /mnt/hugepages \
         -serial telnet::9003,server,nowait > guest_log-3.txt 2>> guest_log-3.txt &
sleep 5
brctl addif br0 tap0
brctl addif br0 tap1
brctl addif br0 tap2

There are some parameters need to understand in QEMU script.

-uuid d0000000-1111-1111-1111-11111aaaaaaa \

this will assign a unique uuid for each VM, k8s needs different VM run with different uuid and MAC address.
It also can use command “uuidgen” to generate different uuid for each VM, but here just for debug so assign a “easy to read” uuid.

-device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \

Assign a special mac address for VM, because for k8s internal communication, use the same mac address will cause problem, please configure different address before run qemu script. It not easy to modify it after first boot (with cloud image)

* Notice, first number for mac address cannot be odd , it would cause problem.

-mem-path /mnt/hugepages \

Tell VM to use hugepage, if were not configure host with hugepage, please remove this line.

Now, please run scripts

bash nic-br.sh
bash run-qemu.sh 

Now, it can use telnet to connect each VM’s console

telnet localhost 9001
# or 
telnet localhost 9002
# or 
telnet localhost 9003

After system booted/initialed, if you cannot login, please “killall qemu-system-aarch64” and follow this article Change Ubuntu Cloud Image Password to change all images’s password.

Somehow, for recent cloud image, it cannot modify password before run qemu script.

Install Kubernetes

There are a lot of article to talk about how to install on Ubuntu, I choice one for reference : Install and Deploy Kubernetes on Ubuntu 18.04 LTS

Below will shortly note install command here.

Install Docker and Kubeadm on every node

sudo apt update 
sudo apt -y upgrade 

# install and enable docker
sudo apt -y install docker.io 
sudo systemctl enable docker 
sudo systemctl start docker 

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add 
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update
sudo apt-get install kubeadm -y

Configure Network IP address (Optional)

Update to you network environment, here we used fix ip address.

Add new record to /etc/hosts

192.168.110.50     k8s-master
192.168.110.60     k8s-worker-node1
192.168.110.70     k8s-worker-node2

Please modify /etc/netplan/50-cloud-init.yam, remove dhcp and add fix address. remember, each node has different IP. Following is only for k8s-master.

network:
     ethernets:
         eth0:
             addresses: [192.168.110.50/16,]
             gateway4: 192.168.1.1
             nameservers:
                 addresses: [8.8.8.8,8.8.4.4]
             match:
                 macaddress: '52:54:00:11:11:11'
             set-name: eth0
     version: 2

Configure Host name

Run following command on master node

sudo hostnamectl set-hostname  k8s-master

Run following command on other nodes

sudo hostnamectl set-hostname   k8s-worker-node1 
sudo hostnamectl set-hostname   k8s-worker-node2

Disable swap (Option)

if image is not from cloud image, default Linux enabled swap function, k8s doesn’t support it, please disable it manually on all nodes and remove swap mount in /etc/fstab.

 sudo swapoff -a 

Configure Kubernetes

Now, we can initial Kubernetes. Run following command

 sudo kubeadm init --pod-network-cidr=10.244.0.0/16 

wait for few minutes, it will show some result like this

Run following command to assign current account has right to access k8s.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run this command on another node(k8s-worker-node1 and k8s-worker-node2).

sudo kubeadm join 192.168.110.50:6443 --token 46c1oj.i00vyfyn027pgew9 \
     --discovery-token-ca-cert-hash sha256:ed05198cc603ae009a862477bacef329a9e9a2bedc4a040bc4731e6d5cf4f48e

Run get nodes command, it will show current status, the master node status is NotReady, but don’t worry, it will become Ready after add pod and CNI (container networking interface) plugin.

ubuntu@k8s-master:~$ kubectl get nodes
 NAME         STATUS     ROLES    AGE     VERSION
 k8s-master   NotReady   master   3m42s   v1.16.0

This action is very import on ARM64 k8s platform, in other document, they told you use flannel or some other document told you used old flannel. all of them not work for me. Please instead flannel with weave net.
(Ref Weave Net for NetworkPolicy and CoreDNS is not starting on Ubuntu 18.04.Bionic Beaver #980)
Run following command to install weave net

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Use the following command to view the status of the network:

Get nodes again.

Great!! it works.

MicroK8s

If you think steps above are too complex and you would like to only run k8s on one machine, microk8s is a good friend for you, can run on ARM64 and very easy to use, just run command

sudo snap install microk8s

And just add a prefix “microk8s.” before k8s command, ex:

microk8s.kubectl get nodes

See, it’s very easy to use. right?

The post Install Kubernetes on ARM64 Ubuntu 18.04 (with QEMU) appeared first on richliu's blog.

Add mac parameter to qemu command

in qemu command line, add mac= parameter to netdev option , ex:

-device virtio-net-device,netdev=net2,mac=52:54:00:56:12:34 \

Modify netplan

edit file /etc/netplan/50-cloud-init.yaml, and add section match, ex

network:
     version: 2
     ethernets:
         eth0:
             dhcp4: true
             match:
                 macaddress: 52:54:00:56:12:34
             set-name: eth0

now, you can shutdown qemu system and restart .

* Notice: if used ubuntu cloud image, please assign mac address before init it.
suppose that cloud image might save mac address some where .

The post Ubuntu 18.04 Change Mac Address on QEMU via Netplan appeared first on richliu's blog.

Hardware : Ampere eMag Server with 128GB Memory
Network Card: Intel i210 Gigabit NIC Card * 2

HOST means commands and instructions on host machines
VM means commands and instructions inside VM

Prepare Environment

Apt source

First, please add deb-src to your database, edit file /etc/apt/source.list and removed all “#” before deb-src, then, run follow commands to update apt database (suppose all commands run with sudo or root in this article)

sed -i 's/# deb-src/deb-src/g' sources.list
apt update

Install Packages

It needs to install relative packages in host

sudo apt install -y dpdk dpdk-dev dpdk-doc 

Only needs to install on HOST.

sudo apt install -y qemu-efi bridge-utils

It will install all necessary dpdk packages in host.

Preload Module

It needs to preload vfio-pci modules by default, run following command

echo "vfio-pci" >> /etc/modules

Configure Line Kernel boot command

It needs to reserve hugepages memory space for Linux kernel, it reserves 32 pages and each pages 1GB, total 32GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=32 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX=”console=tty0 hugepagesz=1GB hugepages=32″

then run foloowing command to update grub.cfg

update-grub

Mount hugetlbfs by default

If wants to use hugepage, it needs to mount hugetlbfs somewhere, we can create a directory, ex: /mnt/hugepages and mount hugetblsfs there by default.

mkdir -p /mnt/hugepages
echo "nodev   /mnt/hugepages                  hugetlbfs       defaults,pagesize=1G    0 0 none" >> /etc/fstab

Bind Network Card for DPDK

It wants to let DPDK use current network card, it needs to bind the network card with dpdk. dpdk-devbind command is a useful command.
List current command

 $ dpdk-devbind --status
 Network devices using kernel driver
 0001:01:00.0 'I210 Gigabit Network Connection 1533' if=enP1p1s0 drv=igb unused=vfio-pci
 0002:01:00.0 'I210 Gigabit Network Connection 1533' if=enP2p1s0 drv=igb unused=vfio-pci
 0006:01:00.0 'I210 Gigabit Network Connection 1533' if=enP6p1s0 drv=igb unused=vfio-pci Active 

There are three network card in the machine, we would like to bind 1:1:0.0 and 2:1:0.0 to vfio-pci driver. please used following command to bind those network cards.

dpdk-devbind -b vfio-pci 0001:01:00.0 0002:01:00.0

and check status again

 $ dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0001:01:00.0 'I210 Gigabit Network Connection 1533' drv=vfio-pci unused=igb
0002:01:00.0 'I210 Gigabit Network Connection 1533' drv=vfio-pci unused=igb

Network devices using kernel driver
============================================
0006:01:00.0 'I210 Gigabit Network Connection 1533' if=enP6p1s0 drv=igb unused=vfio-pci Active 

Both network cards would be under Network devices using DPDK-compatible driver .

Prepare QEMU environment

Set up bridge devices

We already install bridge-utils before, now, it should configure bridge devices.

mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf

Because there is only one network interface on machine, we should use this interface as bridge device, save text below and modify it to fit your environment.

 ORIGNIC=enP6p1s0

 ip addr flush enP6p1s0


 brctl addbr br0
 brctl addif br0 enP6p1s0
 ifconfig br0 up
 ifconfig br0 192.168.110.1 netmask 255.255.0.0
 route add default gw 192.168.1.1


 echo nameserver 8.8.8.8 >> /etc/resolv.conf 

Run script, command brctl show will show how whether it configure property or not.

Prepare QEMU image

Ubuntu 18.04 cloud image is here

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img

and please refer this article to change cloud image’s password
Change Ubuntu Cloud Image Password, before use it, don’t forget to increase cloud image size. 20G is enough for test.

qemu-img resize bionic-server-cloudimg-arm64.img +20G

Next step is to create bios and nvram for qemu image boot.

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Run Qemu

Here is qemu command, save text below and run it

 IMAGE=bionic-server-cloudimg-arm64.img
 sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0.img -pflash flash1.img \
         -device vfio-pci,host=0001:01:00.0 \
         -device vfio-pci,host=0002:01:00.0 \
         -drive file=$IMAGE,if=none,id=disk1 \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0 \
         -mem-path /mnt/hugepages \
         -serial telnet::9001,server,nowait > guest1_log.txt 2>> guest1_log.txt &
 sleep 5
 brctl addif br0 tap0 

-m 8192 : use 8G RAM for VM
–mem-path will use /mnt/hugepages for VM memory

it will create a network device tap0 for bridge device interface between VM and host. and create a telnet server on port 9001, it can use “telnet 9001” to get console, I would like to suggest to use ssh if it can access console.

Remove cloudinit service

it needs to remove cloud-initramfs-copymods, it will copy modules from ramdisk to harddriver, and cloud-init, it’s for some cloud service like aws(?)

removed them would help system runs better.

apt remove cloud-initramfs-copymods cloud-init 

Disable Services

After boot, it needs to disable some default service to improve speed, here is some services can be disabled. (needs run those command with root)

systemctl disable ufw
systemctl disable lxcfs
systemctl disable atd
systemctl disable cron
systemctl disable irqbalance
systemctl disable apparmor
systemctl disable apport
systemctl disable ebtables
systemctl disable grub-common
systemctl disable unattended-upgrades
systemctl disable iscsid

Most import service is irqbalance, without disable it, performance will be drop down.

Now, it can reboot VM now.

Prepare QEMU environment

Prepare QEMU environment and prepare host almost the same, only differences are

1. in QEMU, apt install doesn’t needs to install qemu-efi and bridge-utils
2. in QEMU, before call dpdk-devbind command, it needs to configure enable_unsafe_noiommu_mode for ARM64 platform
3. Hugepage size is different , in HOST it uses 32G, in VM we uses 4G

Enable unsafe noiommu

echo 1 > /sys/module/vfio/parameters/enable_unsafe_*

To permanently enable the no-IOMMU mode, add it to modprobe.d: (ref)

echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf

It also needs to reserve hugepages memory space for Linux kernel for VM, it reserves 4 pages and each pages 1GB, total 4GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=4 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX=”console=tty0 hugepagesz=1GB hugepages=4″

then run foloowing command to update grub.cfg

update-grub

Run DPDK l3fwd

Before run l3fwd, we need to recompile it.

. /usr/share/dpdk/dpdk-sdk-env.sh
make -C /usr/share/dpdk/examples/l3fwd

Bind network interface

dpdk-devbind -b vfio-pci  enp0s1
dpdk-devbind -b vfio-pci  enp0s2

Run l3fwd example

cd /usr/share/dpdk/examples/l3fwd/build
./l3fwd -c 1 -- -p 0x3 --config="(0,0,0),(1,0,0)"

Multi Queue example

./l3fwd -l 1,2,3,4 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2),(0,1,3),(1,1,4)"

Other

There is good tool to get hugepage information

apt install hugepages
hugeadm --pool-list

The post Ubuntu 18.04 ARM64 DPDK in VM (QEMU) appeared first on richliu's blog.

[ 0.000000] cannot allocate crashkernel (size:0x80000000)
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.18.0-25-generic root=UUID=ee8be10d-d649-425d-a807-640751836186 ro console=tty0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M

reserve memory is for RAM disk. root cause might be system cannot allocate enough memory for crash kernel.

Reduce allocate memory size will work

dmesg | grep crash
[ 0.000000] crashkernel reserved: 0x00000000bfe00000 – 0x00000000ffe00000 (1024 MB)
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.18.0-25-generic root=UUID=ee8be10d-d649-425d-a807-640751836186 ro console=tty0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:1024M,high,128G-:16384M

The post ARM64 Kdump cannot allocate crashkernel appeared first on richliu's blog.

一般 change image size 是用這個指令

$ sudo qemu-img resize $UBUNTUVERSION-server-cloudimg-$ARCH.img +50G0
ex:

$ sudo qemu-img resize xenial-server-cloudimg-amd64-uefi1.img +50G

將 qcow2 image format 變成 raw disk

$ sudo qemu-img convert  xenial-server-cloudimg-amd64-uefi1.img xenial-server-cloudimg-amd64-uefi1.raw

之前測 ARM64 17.04 和 17.10 都會自動增加 partition size
但是最近碰到 18.04 似乎不會. (Both X64 and ARM64)

如果碰到的是 16.04 要增加 partition size 有問題, 要用 parted 修改 (似乎 fdisk 和 gdisk 都有問題)
ex:

$ parted xenial-server-cloudimg-amd64-uefi1.raw
(parted) p
Model:  (file)
Disk /work/research/vm/1.raw: 56.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
14      1049kB  5243kB  4194kB                     bios_grub
15      5243kB  116MB   111MB   fat32              boot, esp
 1      116MB   2361MB  2245MB  ext4

(parted) resizepart
Warning: Not all of the space available to /work/research/vm/1.raw appears to be used, you can fix the GPT to use
all of the space (an extra 104857600 blocks) or continue with the current setting?
parted: invalid token: 1

Partition number? 1
End?  [2361MB]? 56G

離開就會儲存

Ubuntu 16.04 的 cloud image 不知道為什麼 cloud-init 會無法結束, 如果不能接觸到 login 那 cloud image 也沒有用.

這時可以用另一個方式移除 cloud-init 相關的套件,
1. 使用 kpartx 將 image 掛到 loop devices 如果沒有, 就先安裝 kpartx 套件

$ sudo apt install kpartx 

2. mount partition (還有 proc, dev, sys)
3. 移除 cloud 相關套件
4. umount partition (要先 umount proc, dev, sys
5. 移除 loop device
6. convert raw file 到 qcow2 格式

mount cloud image partition

$ sudo kpartx -av xenial-server-cloudimg-amd64-uefi1.raw
add map loop0p1 (253:0): 0 109147673 linear 7:0 227328
add map loop0p14 (253:1): 0 8192 linear 7:0 2048
add map loop0p15 (253:2): 0 217088 linear 7:0 10240

loop0p1 就是我們要 mount 的 disk , (註: 有可能因系統不同, 所以會有其他的 device name)

mount disk

$ sudo su
$ mkdir disk
$ mount /dev/mapper/loop0p1 disk 
$ cd disk
$ mount --bind /proc proc
$ mount --bind /dev dev
$ mount --bind /sys sys
$ cd ..
$ chroot disk 

移除 cloud-init 相關的套件

$ dpkg  -l |grep cloud
ii  cloud-guest-utils                0.27-0ubuntu25.1                           all          cloud guest utilities
ii  cloud-init                       18.3-9-g2e62cb8a-0ubuntu1~16.04.2          all          Init scripts for cloud instances
ii  cloud-initramfs-copymods         0.27ubuntu1.5                              all          copy initramfs modules into root filesystem for later use
ii  cloud-initramfs-dyn-netconf      0.27ubuntu1.5                              all          write a network interface file in /run for BOOTIF
ii  ubuntu-cloudimage-keyring        2013.11.11                                 all          GnuPG keys of the Ubuntu Cloud Image builder

$ dpkg -r cloud-guest-utils  cloud-init cloud-initramfs-copymods  ubuntu-server

umount disk

$ exit 
$ cd disk 
$ umount proc
$ umount dev
$ umount sys
$ cd ..
$ umount disk

最後將 raw disk image 轉成 qcow2

$ sudo qemu-img convert  xenial-server-cloudimg-amd64-uefi1.raw xenial-server-cloudimg-amd64-uefi1.img

這樣 16.04 開機就不會卡在 cloud init script 了.

The post Change Ubuntu Cloud Image Size appeared first on richliu's blog.

If want to use it, it needs to add new username and password into it.
and remove some packages.Download cloud Image from ubuntu, ex: download Ubuntu Zesty Cloud Image

wget https://cloud-images.ubuntu.com/zesty/current/zesty-server-cloudimg-arm64.img 

If locate in Taiwan, there is Taiwan mirror site,http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/, it would be fast then cloud-images.ubuntu.com

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/zesty/current/zesty-server-cloudimg-arm64.img

Zesty only has UEFI image, so, there is no -disk1 or -uefi postfix keyword.

If used bionic (Ubuntu 18.04), if you found change password function doesn’t work.
Please shutdown this image (suppose run with QEMU) and run change password command again .

Suppose that something modify password when booting on Ubuntu 18.04.

Install backdoor-image package, this program can help to install new username and password into cloud image.

apt install bzr qemu-utils  
bzr branch lp:~smoser/+junk/backdoor-image  

sudo apt-get install -qy cloud-image-utils git
git clone https://gist.github.com/8c65b8771d5ab1d99c44c285323dfff6.git backdoor-image

Add new password, but first account is not work for me most time, so, I would like to add new username and password. (note: please don’t use this as default password on business purpose. )

cd backdoor-image  
sudo ./backdoor-image --user ubuntu --password ubuntu --password-auth ../zesty-server-cloudimg-arm64.img 
sudo ./backdoor-image --user test --password 123456 --password-auth ../zesty-server-cloudimg-arm64.img 

After boot into virtual machine with this image, it can remove unnecessary packages.

sudo apt remove cloud-initramfs-copymods cloud-init

The post Change Ubuntu Cloud Image Password appeared first on richliu's blog.

我用的是 Ubuntu 17.04, 如果要在別的版本上執行, 會有一點差異

安裝所需要的 Package

$ sudo apt install build-essential pbuilder bc debiandoc-sgml libbogl-dev libc6-pic libslang2-pic libnewt-pic genext2fs mklibs genisoimage dosfstools  tofrodos mtools po4a bf-utf-source fakeroot crash kexec-tools makedumpfile kernel-wedge xsltproc docbook-xml docbook-xsl u-boot-tools xorriso grub-efi-arm64-bin grub-common	

在 /etc/apt/source.list 加上這個來源

deb http://tw.ports.ubuntu.com/ubuntu-ports zesty main/debian-installer
deb http://tw.ports.ubuntu.com/ubuntu-ports zesty-updates main/debian-installer
deb http://tw.ports.ubuntu.com/ubuntu-ports zesty-backports main/debian-installer 
deb http://tw.ports.ubuntu.com/ubuntu-ports zesty-security main/debian-installer

接下來是升級系統

$ sudo apt update
$ sudo apt upgrade 

下載 debian-installer 並且準備好所需要的目錄

$ mkdir build && cd build
$ apt-get source debian-installer
$ tar xvf *.xz
$ cd debian-installer/build
$ mkdir dest

接下來編輯要做的 netboot image version, 修改 config/arm64.cfg, 像我只是要用最新版 kernel 的 boot image.
改版號就可以了. 我是用和系統一樣的版號

KERNELMAJOR = 2.6
#KERNELVERSION = 4.10.0-19-generic
KERNELVERSION = 4.10.0-24-generic

最後一步, 做 netboot image, 看狀況可能很快, 也有可能要載套件花幾分鐘.

$ make build_netboot

完成後的檔案會放在 dest/netboot 內

# ls dest/netboot/
mini.iso  netboot.tar.gz  ubuntu-installer

ref.
Using a custom kernel with the Ubuntu Installer

The post Ubuntu Create Netboot Image appeared first on richliu's blog.

$ debootstrap --verbose --foreign --arch mips jessie ./   http://ftp.tw.debian.org/debian

mips64 只有 mips64le , 沒有 mips64 .

其他的還是看這篇文章就好, 也快八年了.
[ARM][Linux] 建立 ARM 平台的 Debian System

懶得再開一篇, 改一改就好了.
如果是裝 ARM64 Ubuntu 呢? mirror 改到 ports.ubuntu.com 去就可以了, 也有 tw.ports.ubuntu.com 但是我記得是指向國外, 記得 arch 和版本號都要改
Ubuntu 16.04 LTS

$ debootstrap --verbose --foreign --arch arm64 xenial ./ubuntu http://tw.ports.ubuntu.com/ubuntu-ports

Ubuntu 18.04 LTS

$ debootstrap --verbose --foreign --arch arm64 bionic ./ubuntu http://tw.ports.ubuntu.com/ubuntu-ports

Ubuntu 20.04 LTS

$ debootstrap --verbose --foreign --arch arm64 focal ./ubuntu http://tw.ports.ubuntu.com/ubuntu-ports

Ubuntu 22.04 LTS

$ debootstrap --verbose --foreign --arch arm64 jammy ./ubuntu http://tw.ports.ubuntu.com/ubuntu-ports

Debian Stable arm (not ARM64)

debootstrap --verbose --foreign --arch armel stable ./   http://ftp.tw.debian.org/debian

我愛用 NFS 開機, 這是 u-boot NFS 的 example command

setenv bootargs 'console=ttyAMA0,115200n8 earlycon=pl011,0x87e028000000 debug maxcpus=24 rootwait rw ip=dhcp::eth2:off root=/dev/nfs rw nfsroot=10.0.0.30:/nfs/ubuntu1804,v3 coherent_pool=16M'

setenv start 'setenv ethact vnic2 ; dhcp ;  setenv serverip 10.0.0.30 ;  tftpboot $kernel_addr Image ; booti $kernel_addr - $fdtcontroladdr'

run start

如果是安裝的機器本身就是 ARM64 的機器，可以 chroot 到 ubuntu 內
如果是用 NFS 開機, 可以跳過這一個階段，直接執行下面的 run second-stage

cd /ubuntu
mount --bind /proc proc
mount --bind /dev dev
mount --bind /sys sys
cd etc
cp /etc/resolv.conf . 
cd / 
chroot ubuntu bash

執行, 解壓相關的 package (run second-stage)

/debootstrap/debootstrap --second-stage

Ubuntu 16.04 編輯 /etc/apt/source.list 加上

deb http://tw.ports.ubuntu.com/ubuntu-ports xenial main restricted
deb http://tw.ports.ubuntu.com/ubuntu-ports xenial-updates main restricted
deb-src http://tw.ports.ubuntu.com/ubuntu-ports xenial main restricted universe multiverse
deb-src http://tw.ports.ubuntu.com/ubuntu-ports xenial-updates main restricted universe multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports xenial universe
deb http://tw.ports.ubuntu.com/ubuntu-ports xenial-updates universe
deb http://tw.ports.ubuntu.com/ubuntu-ports xenial multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports xenial-updates multiverse

18.04

deb http://tw.ports.ubuntu.com/ubuntu-ports bionic main restricted
deb http://tw.ports.ubuntu.com/ubuntu-ports bionic-updates main restricted
deb-src http://tw.ports.ubuntu.com/ubuntu-ports bionic main restricted universe multiverse
deb-src http://tw.ports.ubuntu.com/ubuntu-ports bionic-updates main restricted universe multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports bionic universe
deb http://tw.ports.ubuntu.com/ubuntu-ports bionic-updates universe
deb http://tw.ports.ubuntu.com/ubuntu-ports bionic multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports bionic-updates multiverse

20.04

deb http://tw.ports.ubuntu.com/ubuntu-ports focal main restricted
deb http://tw.ports.ubuntu.com/ubuntu-ports focal-updates main restricted
deb-src http://tw.ports.ubuntu.com/ubuntu-ports focal main restricted universe
multiverse
deb-src http://tw.ports.ubuntu.com/ubuntu-ports focal-updates main restricted u
niverse multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports focal universe
deb http://tw.ports.ubuntu.com/ubuntu-ports focal-updates universe
deb http://tw.ports.ubuntu.com/ubuntu-ports focal multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports focal-updates multiverse

22.04

deb http://tw.ports.ubuntu.com/ubuntu-ports jammy main restricted
deb http://tw.ports.ubuntu.com/ubuntu-ports jammy-updates main restricted
deb-src http://tw.ports.ubuntu.com/ubuntu-ports jammy main restricted universe multiverse
deb-src http://tw.ports.ubuntu.com/ubuntu-ports jammy-updates main restricted universe multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports jammy universe
deb http://tw.ports.ubuntu.com/ubuntu-ports jammy-updates universe
deb http://tw.ports.ubuntu.com/ubuntu-ports jammy multiverse
deb http://tw.ports.ubuntu.com/ubuntu-ports jammy-updates multiverse

加上 apt cache (如果你有內部的 proxy server)

$ echo "Acquire::http::Proxy \"http://192.168.1.254:3142\";" | tee -a /etc/apt/apt.conf

更新套件庫

$ apt update
$ apt -y upgrade

安裝必要的程式 (註：註 net-tools 過時了, 可以我還是習慣舊指令)

$ apt install -y vim openssh-server tmux btrfs-tools net-tools sudo wget resolvconf-admin

# 20.04/22.04 btrfs 改叫 btrfs-progs
$ apt install -y vim openssh-server tmux btrfs-progs net-tools sudo wget resolvconf-admin

Add User

$ adduser [username]

順便改 root 的密碼

$ passwd root

sudo 加上無密碼 請參考這個連結

註: 18.04 可能要先設定密碼之後重開機一次才能, 更新系統環境變數, 安裝套件以便之後的設定

註2: 18.04 如果時間差太多會 update 失敗, 要先調整時間到今天
先裝 nfs-common 增加 nfs access 的速度

$ apt install -y nfs-common

備用, 17.04 後似乎都會用 resolvconf 當 DNS service, 但是不知道為什麼不會取用 DHCP 得到的 DNS

$ systemctl disable systemd-resolved.service
$ service systemd-resolved stop
$ echo "nameserver 8.8.8.8" > /etc/resolv.conf

這樣看看會不會正常

編輯 /etc/hostname
取一個合適的名字, 預設是用執行 debootstrap 的主機名稱

編輯 /etc/network/interface

最後要產生 utf-8 的 locale 檔案

$ sudo locale-gen en_US.UTF-8

The post MIPS/ARM64 Debian/Ubuntu debootstrap appeared first on richliu's blog.

改寫一下, 這是講 ubuntu clouding image , 不過一般 Desktop 應該也適用

At the end of /etc/sudoers there is what I thought was just a comment:
#includedir /etc/sudoers.d

You should edit this file using visudo. The following command will let you edit the correct file with visudo.

$ sudo visudo -f /etc/sudoers.d/90-cloudimg-ubuntu 

And add a line like:

[username] ALL=(ALL) NOPASSWD:ALL 

At the end.

http://askubuntu.com/questions/192050/how-to-run-sudo-command-with-no-password

The post Ubuntu Sudo Nopasswd appeared first on richliu's blog.

GPIO(0)-NODE(0): Vector:100 address :801000000050 irq:33
GPIO(0)-NODE(0): Vector:101 address :801000000058 irq:33
Using TWSI func = 0x48
UEFI Interactive Shell v2.1
choice 3 shell

ThunderX Cavium UEFI install ubuntu CD

enter shell mode and put usb cd-rom .

GPIO(0)-NODE(0): Vector:100 address :801000000050 irq:33
GPIO(0)-NODE(0): Vector:101 address :801000000058 irq:33
Using TWSI func = 0x48
UEFI Interactive Shell v2.1
EDK II
UEFI v2.40 (Cavium Thunder cn88xx EFI ThunderX-Firmware-Release-1.22.11-0-g06c3f37 Sep)
Mapping table
FS0: Alias(s):CD36a0b0a:;BLK1:
PciRoot(0x0)/Pci(0x11,0x0)/USB(0x0,0x0)/USB(0x1,0x0)/CDROM(0x0)
BLK2: Alias(s):      
VenHw(25E45362-4074-46DC-88A0-79D6A23F3C9D)
BLK0: Alias(s):                     
PciRoot(0x0)/Pci(0x11,0x0)/USB(0x0,0x0)/USB(0x1,0x0)

Press ESC in 3 seconds to skip startup.nsh or any other key to continue.
Shell> fs0:                                   
FS0:\> ls      

run
/boot/grub/arm64-efi

The post Cavium ThunderX Install Ubuntu CD appeared first on richliu's blog.

首先, Kubutun 15.04 支援 UEFI, 所以這部份安裝完後可以正常 Dual Boot 到 Windows/Linux.
Linux 對於 Broadwell i7-5775C 的支援, 大家的講法不一, 目前看起來應該是這個 URL 比較常見
Working Around The Intel Core i7 5775C Broadwell Stability Issue On Linux
圖中貼了一個 crash , 我也在舊版的 Kernel 見過, 文章表示 enable CPU OC Fix mode 就可以解決這個問題.

我的血淚史是: ASUS 在我裝完機之後又新出一個 BIOS, 這個 BIOS 解決了 i7 5775 一直會出現 Machine Exception Check 的問題.
這個問題只要是在機器上執行 Linux 就會出現, 而且如果是裝 Virtual Machine 時, 再裝 Linux 也會出現 (真是神祕啊).
還沒有安裝新的 BIOS 之前, 這問題搞倒我了, 最後還是靠上面那一篇的類似解決方法才可以正常完成安裝(ASUS BIOS 沒有這個選項, 所以我是關 SpeedStep Does Linux run correctly with SpeedStep on Broadwell?)而且還是不穩定.
然後我在將近一個月後之才看到有新的 BIOS Release.
ASUS 你不可以早點做嗎?

另一個問題就是上面那篇文章的錯誤訊息, 我本來想回報到 Linux kernel team bug system , 但是想想是不是應該先換到最新版的 Linux kernel 再試呢?
Kubuntu 15.04 最新的 Kernel 是 3.19.0-26-generic
這時候就要找 Ubuntu mainline kernel 了, 客倌看到這邊應該會想, 要我編 Linux kernel , 殺了我吧.

沒錯, 不過不用殺頭, 我自己也很久沒有下去看/編 Kernel 的東西了, 中間有個連結 Upstream kernels archive http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D 點下去就對了.

會看到很多目錄, 我是選最新的 v4.2-rc7-unstable (註: 一直會更新, 所以不見得最新的會和我一樣)
下載 generic 的就好了, 我是 x64 的系統, 所以下的是 amd64 image .
linux-image-4.2.0-040200rc7-generic_4.2.0-040200rc7.201508162030_amd64.deb
linux-headers-4.2.0-040200rc7-generic_4.2.0-040200rc7.201508162030_i386.deb
下載完之後再用 root 執行 dpkg -i 就好了
# sudo dpkg -i linux-image-4.2.0-040200rc7-generic_4.2.0-040200rc7.201508162030_amd64.deb

換了 Linux kernel 4.2RC 之後, Working Around The Intel Core i7 5775C Broadwell Stability Issue On Linux 這個問題就沒有再出現過了.

以上解決方案有錯, 仍然會出現 MCE.
根據 Linux Kernel Bug Report 上的說法,
在 bootup 時的 kernel 參數加上 “processor.max_cstate=0 intel_idle.max_cstate=0 idle=poll” 就可以暫時避開這個問題.
以上這個暫時解決方案會讓 CPU 滿載, 所以建議可以在安裝的時候使用.

如果要穩定一點的解決方案, 可以換到最新版的 Microcode , 原因可能是 Intel 在 Broadwell 仍然未解 TSX 的 Bug
而且也沒有更新 Microcode(目前最新是今年一月), 所以各家 Linux Distributions 也無法更新 Microcode 解決這個問題.
如果 glibc compile 時並未打開 TSX support, 看起來不會踏到這個 BUG
但是如果是有 enable TSX support 的 Linux Distribution , 就會踏到這個 BUG

Ubuntu 下的解決方式如下
1) 下載 iucode-tool
$ sudo apt-get install iucode-tool
2) 下載新版的 Microcode
$ git clone https://github.com/bgw/bdw-ucode-update-tool.git
$ cd bdw-ucode-update-tool
$ sudo ./install.sh –persist-debian
3) 驗證, microcode 版本是 0x13
$ cat /proc/cpuinfo | grep microcode
microcode : 0x13

這還是有缺點,
1) 新裝系統
2) 更新 Kernel
都還是要跑一次這樣的流程, 最好還是跟主機板廠商連絡請他們更新 BIOS .
這邊我就要抱怨 ASUS 了, 搞了一個多月來來回回還是在講同一個問題, 至現在 (11/9) 為止仍然沒有更新 BIOS.

更新完後系統穩定, 並未碰到當機事件了.

後記, 個人覺得以 SoC Errata 的處理, Intel 非常槽糕, 我知道新功能的確可能會有問題.
但是出廠前 Linux/Windows 都應該要測試過才會出門吧.
沒有想到只有 Windows OK.

我在以前根本沒有想到買新 CPU 要先考慮在 Linux 上會不會有問題.
真是 WTF

ref.
Bug 103351 – Machine check exception on Broadwell quad-core with SpeedStep enabled
[haswell] Unable obtain Package Cstates C3 or lower

The post Kubuntu 15.04 + Broadwell / Intel i7-5775C 的問題 appeared first on richliu's blog.

目前看到寫的最簡單的就是這一篇了.

How to Install Ubuntu 14.04 with RAID 1 using desktop installer?

apt-get install mdadm
mdadm --create /dev/md0 --bitmap=internal --level=1 -n 2 /dev/sda1 missing
fdisk /dev/md0

會產生 /dev/md0p1 /dev/md0p2 /dev/md0p3

用 try ubuntu 在 console 下這個指令安裝, 要做點小手腳

ubiquity -b

安裝的時候要選 /dev/md0p1 /dev/md0p2 /dev/md0p3 不要選別的.

安裝完之後, 要重新安裝 grub

sudo -s
mount /dev/md0p1 /mnt
mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -o bind /sys /mnt/sys
mount -o bind /proc /mnt/proc
chroot /mnt
apt-get install mdadm
nano /etc/grub.d/10_linux  # change quick_boot to 0
grub-install /dev/sda
grub-install /dev/sdb
update-grub
exit

如果 /etc/grub.d/10_linux quick_boot 沒有設成 0
開機會出現

error:  Diskfilter writes are not supported

The post kubuntu 14.04 Desktop Install on Raid Disk appeared first on richliu's blog.

頻道長輩因為 ASUS EeePC 的 4G SSD 有點問題, 就送出這一台 EeePC 701 4G Sure.
這一台沒有可以擴充的介面, 要改裝要拆機才可以改, 不是很方便.

還好還有一個 SD Card 插槽, 可以丟一張 8G SD Card 進去, 不過該 SD Card 插槽的介面速度真是悲劇.. 超慢的.

昨天拿到先灌了 Windows XP 進去, 看起來沒有大問題.
(註: 隔天跑到一半還是爛掉, 原因不明, 但是 memory test 都是 pass 的).
但是灌 Ubuntu Linux 就不是這麼容易, 這顆 CPU 不支援 PAE, 所以 Ubuntu 12.04 不能用, 如果使用安裝光碟, 會有 CPU 不支援的字樣.

網路上有人說用 mini ISO 可以灌成功, 不過我怎麼試都會在 Install Package 那邊失敗.
還想要用舊版的 Ubuntu 8.04 灌完再升級, 不過還是失敗 (囧)

最後找到有文章(What’s the meaning of “-pae” at the end of kernel version?)提到 Lubuntu 仍然使用 non-PAE kernel 做為開機光碟, 去下載 Lubuntu 12.04, 果然可以順利解決這個問題.
Lubuntu 預設使用 LXDE 桌面系統, 短小精悍, 而且果然適合 EeePC 這種超級老系統, 我也就懶得換 Desktop 了.
缺點就是, Install Software 不是設計給 800×480 這種解析度用的, 有點難操作.

裝完之後不知道為什麼, grub 沒有裝進去, 會開機失敗, 需要進救援模式
# mount /dev/sdb1 /mnt
# cd /mnt
# mount –bind /proc proc
# mount –bind /dev dev
# mount –bind /sys sys
# cd /
# chroot /mnt
# apt-get install grub2
# grub-install /dev/sda
grub2 不需要像 grub1.x 用 grub + command 模式設定(比較穩定?)
似乎改善很多.

開進系統大概要二~三分鐘吧 (看在用 SD Card 的份上, 算了)

因為我是要拿來測 Sierra Wireless AirCard 320U 的.
首先先按照這份文件(Can I use a Sierra Wireless Modem on Linux Machines (direct IP modems)?)下載 Driver v3.2_1740 按照文內的方式安裝,

Ubuntu 12.04 內建 Driver, 插入就可以抓得到, 如果沒有出現 wwan0 interface , 可以重開機再試一下.

接下來就用 minicom 連到 /dev/ttyUSB3 介面, 下 at command (這真古老的命令, 以前玩 modem 的時候下很多)
$ ati5
$ at+cgdcont=1,”IP”,”Aeroflex.com”
*註: 這是我的測試 SIM Card
$ at!pcstate=1
$ at!scact=1,1
這樣兩個燈就會亮了, 表示己經連上 EPC. 但是這時還沒有拿到 IP.
在 Linux Console 下,
# dhclient wwan0

這樣我的 EeePC 就可以走 LTE 上網了….

Update:

在 BIOS 將 OS Installed 選項改成 Finished 似乎會比較快?

另外我現在改用 wvdial 套件, 可以修改 /etc/wvdial.conf 改成如下的設定
[TEXT]
[Dialer Defaults]
Init1 = at+cgdcont=1,”IP”,”Aeroflex.com”
Init2 = at!pcstate=1
Init3 = at!scact=1,1
Baud = 115200
; New PPPD = yes
Modem = /dev/ttyUSB3
ISDN = 0
[/TEXT]

再執行
# wvdial
就可以連上了, 若是 USB LTE Dongle 上二個燈號都亮的時侯, 要手動啟動 wwan0 interface, 再手動執行 dhclient 抓 IP , 這樣就可以了.
# ifconfig wwan0 up
# dhclient wwan0

The post ASUS EeePC 701 4GSurf + lubuntu + Sierra AirCard 320U appeared first on richliu's blog.

The post Ubuntu 連 PPTP Server 的設定. appeared first on richliu's blog.

Ubuntu 預設的 xL2TPD 似乎不會使用 L2TP Kernel module, 可能要下載最新版的 xL2TPD source code 才可以.
有成功和沒有成功 performance 差距極大.

OpenL2TPD 一定是 kernel module, 所以有成功就是有上 kernel, 沒有成功就是爛掉.

1. 安裝套件
[BASH]
# sudo apt-get install flex bison libreadline-dev rpcbind
# sudo apt-get install ppp-dev
[/BASH]

2. 下載 source code
從 ftp://ftp.openl2tp.org/releases/openl2tp-1.8/openl2tp-1.8.tar.gz 下載 OpenL2TPD 的 source code
從 http://sourceforge.net/projects/openl2tp/files/ippool/ 下載 ippoold source code

3. 解壓 openl2tpd
3.1 修改 Makefile
CFLAGS= 這邊移掉 -Wall -Werror (也可以 search Wall 後再移掉), 要不然 compile 時會有錯誤訊息

3.2 修改 usl/Makefile
CFLAGS= 這邊移掉 -Wall -Werror

3.3 安裝
# make
# make install

4. 解壓 ippoold
4.1 修改 usl/Makefile
CFLAGS= 這邊移掉 -Wall -Werror

4.2 修改 usl/usl_timer.c
加上
[TEXT]
#include [/TEXT]

4.3 修改 Makefile
PPPD_VERSION= 2.4.5 ==> 改成你系統上的版本, 可以用 # pppd -v 看版本
以下這幾行加上 -fPIC
[TEXT]
$(IPPOOL_RPC_STEM)_xdr.o: $(IPPOOL_RPC_STEM)_xdr.c
$(CC) -I. -MMD -g -c -w -fPIC $(CPPFLAGS) $(CFLAGS.optimize) $< $(IPPOOL_RPC_STEM)_client.o: $(IPPOOL_RPC_STEM)_client.c $(CC) -I. -MMD -g -c -w -fPIC $(CPPFLAGS) $(CFLAGS.optimize) $< $(IPPOOL_RPC_STEM)_server.o: $(IPPOOL_RPC_STEM)_server.c $(CC) -I. -MMD -g -c -w -fPIC $(CPPFLAGS) $(CFLAGS.optimize) $< [/TEXT] 4.4 安裝 # make # make install 5. 設定 5.1 載入 L2TP kernel module # modprobe pppol2tp -> 舊版 Linux Kernel (沒試過)
# modprobe l2tp_ppp -> 新版 Linux kernel L2TP module.

5.2 修改 /etc/ippool.conf -> 這個檔對我無效, 原因不明
加入以下文字
[TEXT]
pool create pool_name=default first_addr=10.5.1.1 num_addrs=100
[/TEXT]

5.3 修改 /etc/openl2tpd.conf 加上
[TEXT]
ppp profile modify profile_name=default \
ip_pool_name=default
[/TEXT]

5.4 修改 /etc/ppp/chap-secrets, 加上
[TEXT]
test * “test” *
[/TEXT]

5.5 copy init.d file
[BASH]
# cp ippool-1.3/debian/init.d /etc/init.d/ippoold
# cp openl2tp-1.8/debian/init.d /etc/init.d/openl2tpd
# chmod +x /etc/init.d/ippoold
# chmod +x /etc/init.d/openl2tpd
# /etc/init.d/ippoold start
# /etc/init.d/openl2tpd start
[/BASH]

5.6 設定 ippoold , ippoold 和 OpenL2TPD 都是靠 RPC 做通訊, 所以可以執行 ippoolconf 去設定系統
# ippoolconfig
輸入 5.2 的文字
[TEXT]
pool create pool_name=default first_addr=10.5.1.1 num_addrs=100
[/TEXT]
然後檢查結果
[TEXT]
pool show pool_name=default
[/TEXT]

6. Client 端設定, 安裝和 Server 端一樣, 只是不需要安裝 ippoold, 編輯 /etc/openl2tpd.conf
加上以下文字 .
[TEXT]
ppp profile modify profile_name=default \
auth_eap=no auth_mschapv1=no auth_mschapv2=no mtu=1460

tunnel create tunnel_name=katalix dest_ipaddr=10.1.1.135 \
persist=yes

session create tunnel_name=katalix \
session_name=katalix \
user_name=test \
user_password=test
[/TEXT]
dest_ipaddr=10.1.1.135 是對方 IP
user_name=test user_password=test 是帳號和密碼.

這邊提醒一下, mtu 要設 1460.

7. Debug
如果連線有問題, 兩邊都可以將 daemon 關掉, 直接下指令 openl2tpd -Df 看錯誤訊息
大部份可能是
1. RPC 程式沒有啟動
2. ippoold 有問題, 沒有分配 IP
3. username/password 錯誤

8. openl2tp: unable to register RPC program
如果執行時碰到這個錯誤訊息, 改用 portmap 套件看看
[BASH]
# sudo apt-get install portmap
[/BASH]

Ref.
1. http://www.openl2tp.org/
2. 搭建openl2tpd服务器
3. http://wiki.steneteg.org/index.php/Openl2tp

The post Ubuntu 裝 OpenL2TPD 的問題 appeared first on richliu's blog.

[TEXT]
2012-07-18T11:33:31.595+08:00| vthread-3| I120: Building module vmnet.
2012-07-18T11:33:31.595+08:00| vthread-3| I120: Extracting the sources of the vmnet module.
2012-07-18T11:33:31.614+08:00| vthread-3| I120: Building module with command: /usr/bin/make -j -C /tmp/vmware-root/modules/vmnet-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/3.2.0-26-generic-pae/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.6
2012-07-18T11:33:33.216+08:00| vthread-3| I120: Failed to compile module vmnet!
2012-07-18T11:33:33.220+08:00| vthread-3| I120: Trying to find a suitable PBM set for kernel 3.2.0-26-generic-pae.
2012-07-18T11:33:33.220+08:00| vthread-3| I120: Validating path /lib/modules/3.2.0-26-generic-pae/build/include for kernel release 3.2.0-26-generic-pae
[/TEXT]

到 VMware Workstation 8.0.2 7.1.5 / Player 4.0.2 3.1.5 fix for linux kernel 3.2 and 3.3下載 Patch, 解開之後將檔案 patch-modules_3.2.0.sh 內的
[TEXT]
vmreqver=8.0.2
[/TEXT]
改成
[TEXT]
vmreqver=8.0.4
[/TEXT]
再執行它就可以了
[BASH]
# ./patch-modules_3.2.0.sh
[/BASH]

The post Ubuntu 12.04 LTS + VMware 8.04 的問題. appeared first on richliu's blog.

10.04 的用戶

1. edit /etc/update-manager/release-upgrades and set Prompt=lts
2. Run “do-release-upgrade -d” in a terminal

ref.
http://ubuntuforums.org/showthread.php?t=1980839&page=2

The post Ubuntu 10.04 Upgrade to 12.04 LTS 的問題 appeared first on richliu's blog.