2021/08/31 Updated : No more need Ubuntu ISO image, ARM64 EFI minimal ISO can be download from official website: https://www.gentoo.org/downloads/

Some ARM64 platform support UEFI bios system, like Ampere Computing Altea or Marvell ThunderX.
Recently Linux like CentOS or Ubuntu can support both system, but not for Gentoo ARM64 project,
if you want to install Gentoo on ARM64 bare system, it needs some trick skill.

Install System

Gentoo ARM64 install, it needs to relay on other OS’s installation software.
I were using Ubuntu/Debain net-inst to install system, below is my nvme driver partition table.

/dev/nvme0n1p1 (bootloader/efi) 500MB for EFI only
/dev/nvme0n1p2 / 800G for Gentoo system
/dev/nvme0n1p3 / 90G for Ubuntu system

Suggest to keep a ubuntu system for booting, it can be rescue system when fail to boot Gentoo system.

After Ubuntu/Debian installed system, it can go back to installation menu, there is a execute shell on menu, run it.

Decompress Gentoo Stage3 File.

Most challenge thing is, Ubuntu/Debian installation rootfs, there is no unbzip function, that’s means it needs to use installed disk and used the Ubuntu system’s bzip2. below is my command to untar it.

# use gentoo iso cdrom boot
# mount /dev/nvme0n1p3 /mnt/gentoo
# cd /mnt/gentoo
# wget https://bouncer.gentoo.org/fetch/root/all/releases/arm64/autobuilds/20210829T233646Z/stage3-arm64-systemd-20210829T233646Z.tar.xz
# tar xvf stage3-arm64-systemd-20210829T233646Z.tar.xz
# mount --bind /proc proc
# mount --bind /dev dev
# mount --bind /sys sys
# mount --bind /sys/firmware/efi/efivars/ ./sys/firmware/efi/efivars/
# cp /etc/resolv.conf /mnt/gentoo/etc/
# cd ..
# chroot gentoo 

>>>> chroot to ubuntu 
# mkdir  -p /mnt/gentoo
# mount /dev/nvme0n1p2 /mnt/gentoo 
# cd /mnt/gentoo 
# wget http://distfiles.gentoo.org/experimental/arm64/stage3-arm64-systemd-20190925.tar.bz2
# tar xvzf stage3-arm64-systemd-20190925.tar.bz2
# mount --bind /proc proc
# mount --bind /dev dev
# mount --bind /sys sys
# cp /etc/resolv.conf /mnt/gentoo/etc/
# cd ..
# chroot gentoo

Yaa.. We got Gentoo file system .

Gentoo Kernel

Suppose that reader know how to configure hostname , fstab, emerge –sync … etc.

Next step is to install Gentoo kernel, I will show how to use genkernel to generate workable kernel for ARM64 system

*Note: Some install command might need to unmask or add accept keyword, will ignore that, please try add “–autounmask-write” and run command “etc-update” to update it.

# mkdir -p /boot/efi
# mount /dev/nvme0n1p1 /boot/efi
# echo 'GRUB_PLATFORMS="efi-64"' >> /etc/portage/make.conf
# emerge --ask sys-boot/grub:2
# emerge efibootmgr
# grub-install --target=arm64-efi --efi-directory=/boot/efi --bootloader-id=Gentoo

If have problem on the stage, and it shows cannot find /dev/md0, it needs to emerge mdadm package.

Compile Linux Kernel

# emerge gentoo-sources 
# emerge genkernel
# genkernel --menuconfig all

genkernel will let you choice kernel configure before compile it. Please enable those options, Gentoo kernel didn’t enable ARM’s serial port by default.

Device Drivers  --->
	Character devices  --->
		Serial drivers  --->
			 <*> ARM AMBA PL010 serial port support
			[*]   Support for console on AMBA serial port
			 <*> ARM AMBA PL011 serial port support
			[*]   Support for console on AMBA serial port
                        [*] Early console using ARM semihosting

Save it and quit.

Next step is edit /etc/default/grub and add “console==ttyAMA0” to /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyAMA0"

Update grub.conf

grub-mkconfig -o /boot/grub/grub.cfg

Check UEFI Boot Priority

Run efibootmgr and get boot priority

# efibootmgr
BootCurrent: 0003
Timeout: 5 seconds
BootOrder: 0003,0001,0002,0004
Boot0001* ubuntu
Boot0002* UEFI: PXE IP4 Intel(R) Ethernet Server Adapter I210-T1
Boot0003* Gentoo
Boot0004* UEFI: Built-in EFI Shell

It can see current boot and boot order here , if Gentoo is not first priority, it can use option “-o” to update it.

Other

if you choice systemd as your service manager, after install openssh, please run command below to activate it and as default service.

# systemctl enable sshd.service
# systemctl start sshd.service
# systemctl status sshd.service

Conclusion

I have write down my note here, good luck to you if you have the same question then get this webpage.

The post Gentoo ARM64 UEFI Install Guide appeared first on richliu's blog.

Ubuntu

It can refer to those three articles to create a new cloud image from Ubuntu server image, remember to change ARM64 image to x86_64 image .

Change Ubuntu Cloud Image Size (if you choice ubuntu 16.04 as your image)
Change Ubuntu Cloud Image Password
Ubuntu Cloud Image Taiwan Mirror Site

Install Qemu and setup bridge devices

apt install -y ovmf qemu bridge-utils
mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf
echo "allow virbr0" >> /etc/qemu/bridge.conf

Write this script to a file and run it.

SERVERFILE=xenial-server-cloudimg-amd64-uefi1.img
VERSION=`uname -r`

sudo qemu-system-x86_64 -name vm1 \
        -machine pc-i440fx-xenial,usb=off \
        -cpu qemu64 -m 8192 \
        -smp 12,sockets=1,cores=12,threads=1 \
        -nographic -nodefaults \
        -bios /usr/share/ovmf/OVMF.fd \
        -virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 \
        -drive file=$SERVERFILE  \
        -net nic,macaddr=$macaddr \
        -net tap,ifname=tap0 \
        -serial telnet::9001,server,nowait > guest1_log.txt 2>&1 &

sleep 5

brctl addif br0 tap0

it can use command
qemu-system-x86_64 –machine help
and
qemu-system-x86_64 –cpu help
to get all machine and cpu options.

it can change cpu and machine type here.

network would be bridge on tap0

below is good option to share folder between host and VM, below command will share /home/work with vm.

option 
--virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 

it can put this line to /etc/fstab to mount host’s folder to somewhere, in this case, just put the same directory like host does.

host0   /home/work   9p      trans=virtio,version=9p2000.L,user,umask=000   0 0

CentOS

CentOS document was written on 2021, but Ubuntu document on 2019, there is some difference between it. but that’s ok, I believe both ways are works.

Compile QEMU

CentOS default application doesn’t support run x86_64 code on ARM64 server, it needs to compile manually.

Please download qemu and compile it.

# Install CentOS Packages
yum config-manager --set-enabled powertools
yum install -y xorg-x11-xauth python3 ninja-build
yum install -y glib2-devel libmount-devel
yum install -y git glib2-devel libfdt-devel pixman-devel zlib-devel

# Download QEMU Source Code
wget https://download.qemu.org/qemu-5.2.0.tar.xz
# Decompress it
tar xvf qemu-5.2.0.tar.xz
# 
cd qemu-5.2.0
./configure --target-list="x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user"
make -j `nproc`
make install 

Then, qemu will be installed on /usr/local/bin

Download UEFI Image

It needs UEFI Image for x86_64 image boot up. It needs OVMF’s tianocore (Open Source UEFI BIOS) to boot up. And we can download kraxel as prebuild images.

# Install Kraxel into CentOS repos
cd /etc/yum.repos.d/
wget https://www.kraxel.org/repos/firmware.repo
yum install edk2.git-aarch64 edk2.git-ovmf-x64
cd <your vm directory>
cp /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd .
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd .

Prepare Bridge Devices

It needs bridge devices to connect to internet, default qemu bridge device name is virbr0 , due to CentOS doesn’t support brctl, follow this document(3 Ways to Create a Network Bridge in RHEL/CentOS 8) to create a CentOS bridge devices. Below is my experiment script to create bridge device, this setting will keep in system, reboot will not be reset.

nmcli conn add type bridge con-name virbr0 ifname virbr0

killall dhclient
ip addr flush enP7p2s0
nmcli conn modify virbr0 ipv4.addresses '192.168.3.191/16'
nmcli conn modify br0 ipv4.gateway '192.168.1.1'
nmcli conn modify br0 ipv4.dns '8.8.8.8'
nmcli conn modify br0 ipv4.method manual
nmcli conn add type ethernet slave-type bridge con-name bridge-br0 ifname enP7p2s0 master virbr0
nmcli conn up virbr0
nmcli conn up bridge-br0

nmcli conn show  --active

# Allow QEMU to access virbr0
mkdir -p /etc/qemu
echo "allow virbr0" >> /etc/qemu/bridge.conf

Run Qemu

It can refer above to prepare cloud image, I prefer to use ubuntu cloud image, but have not try CentOS image.

Remember Ubuntu 18.04 image won’t auto increase image size, please follow this Change Ubuntu Cloud Image Size to modify image size

qemu-system-x86_64 -m 8192 -smp 16 -nographic \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE-pure-efi.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=OVMF_VARS-pure-efi.fd \

        -hdc bionic-server-cloudimg-amd64.img \
        -netdev bridge,br=virbr0,id=net0 \
        -device virtio-net,netdev=net0,mac=00:11:22:33:44:55 \
        -serial telnet::9001,server,nowait > vmlog.txt 2>> vmlog.txt &

x86_64 application

Sometimes we don’t need to run whole VM to run some application just for x86_64, it can run user mode application directly. I am a lazy guy, so just reuse cloud image as x86_64 Linux library.

mkdir -p /home/amd64
qemu-img convert  bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.raw
losetup /dev/loop0 bionic-server-cloudimg-amd64.raw
kpartx -a /dev/loop0
mount /dev/mapper/loop0p1 /home/amd64

Umount Image

umount /home/amd64
losetup -d /dev/loop0

Run x86_64 only application

qemu-x86_64 -L /home/amd64/ bootutil64e
qemu-x86_64 -L /home/amd64/ nvmupdate64e -l

Unsolved Problem

Cannot find way to update INTEL NIC’s NVM

Tried vfio and application mode, vfio cannot pass iommu to host, it will fail.

User space application will cause mmap fail.

If you know how to update Intel NIC’s NVM, welcome to leave a message for me.

QEMU –static problem

Add –static parameter will cause some problem, cannot solve it right now.

./configure –target-list=”x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user” –static
===

ERROR: sizeof(size_t) doesn’t match GLIB_SIZEOF_SIZE_T.

    You probably need to set PKG_CONFIG_LIBDIR

    to point to the right pkg-config files for your    build target

The post QEMU Run x86_64/AMD64 VM on ARM64/AARCH64 appeared first on richliu's blog.

no valid host was found. There are not enough hosts available.
Code
500
Details
File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", 
line 526, in build_instances context, request_spec, filter_properties) File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", 
line 597, in _schedule_instances hosts = self.scheduler_client.select_destinations(context, spec_obj) File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", 
line 371, in wrapped return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", 
line 51, in select_destinations return self.queryclient.select_destinations(context, spec_obj) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", 
line 37, in __run_method return getattr(self.instance, __name)(*args, **kwargs) 
File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", 
line 32, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj) 
File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", 
line 129, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", 
line 169, in call retry=self.retry) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", 
line 97, in _send timeout=timeout, retry=retry) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 458, in send retry=retry) File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 449, in _send raise result

Possible reason

* if using cloud uefi image, it needs to install qemu-efi package in computer node, don’t know why it won’t install it automatically .
$ apt install qemu-efi
* when add new UEFI image, it needs to add new meta data parameter in Horization
hw_firmware_type=uefi

if see

2017-08-01 14:02:13.612 2409 ERROR nova.compute.manager [instance: b334f0b3-ca9a-4f9a-b488-532dce00265a] libvirtError: unsupported configuration: logfile not supported in this QEMU binary
2017-08-01 14:02:13.612 2409 ERROR nova.compute.manager [instance: b334f0b3-ca9a-4f9a-b488-532dce00265a]
# Use the "logd" backend for handling stdout/stderr from QEMU processes.

Reason
* Libvirt later version support new function, logd, it’s a new console log function, but ARM64 doesn’t support it. workaround solution is to disable it

Edit file /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py, find

MIN_LIBVIRT_VIRTLOGD = (1, 3, 3)
MIN_QEMU_VIRTLOGD = (2, 7, 0)

Replaced it with

MIN_LIBVIRT_VIRTLOGD = (9, 9, 9)
MIN_QEMU_VIRTLOGD = (9, 9, 9)

it can avoid to call libvirt logd function.

rm -f /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.pyc, make sure it will call .py
then, reboot service (or system)

Ref.
http://dozencloud.org/

The post No valid host was found. There are not enough hosts available appeared first on richliu's blog.