System Specifications

Nvidia Driver: NVIDIA-Linux-aarch64-570.86.16.run
Host: Ampere Altra + ALTRAD8UD
Host OS: Ubuntu 22.04 with HWE kernel (6.8)
Guest OS: Ubuntu 22.04 (ubuntu-22.04-live-server-arm64.iso)
GPU: Nvidia RTX 4080 16GB

Assumptions

1. You are familiar with Ubuntu and its basic commands.
2. You have experience using Virt-manager.
3. All commands are executed as the root user.

If anything is unclear, you can refer to external resources for additional guidance.

Host Configuration

Enable IOMMU

To enable IOMMU, you need to enable the SR-IOV option in the BIOS and verify whether the Linux kernel has IOMMU enabled by default.

You can check if IOMMU is enabled by running:

$ dmesg | grep -i iommu

Example output:

[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=6b78fa89-a575-432d-a445-1497c3467214 ro iommu=on
[    0.000000] Unknown kernel command line parameters "BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic iommu=on", will be passed to user space.
[   11.561684] iommu: Default domain type: Translated
[   11.566470] iommu: DMA domain TLB invalidation policy: strict mode

If IOMMU is not enabled, add iommu=on to the Linux kernel boot parameters:

$ vim /etc/default/grub 

Modify the line:

GRUB_CMDLINE_LINUX_DEFAULT="iommu=on"

Then update GRUB and reboot:

$ update-grub2  
$ reboot  

Additionally, enable SR-IOV in the BIOS. The exact location of this setting varies depending on the BIOS, but it is typically found under the PCIe subsystem or related options.

Upgrade Host to HWE Kernel

I recommend using the Hardware Enablement (HWE) kernel on the host. While I’m unsure if the regular kernel works, the HWE kernel has been reliable in my experience. Install it with:

sudo apt install linux-generic-hwe-22.04

Configure VFIO on Host

The VM relies on the VFIO driver for GPU passthrough. To configure VFIO, you need to pass the PCIe device information to the VFIO driver.

First, identify the GPU’s PCIe device IDs:

$ lspci -nn

Example output:

0005:01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2704] (rev a1)
0005:01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bb] (rev a1)

Here, 10de:2704 is the GPU’s PCIe device ID, and 10de:22bb is the audio device ID. At a minimum, you need to pass through the GPU device.

Next, edit the VFIO configuration file to include these IDs:

$ vim /etc/modprobe.d/vfio.conf

Add the following line:

options vfio-pci ids=10de:2704,10de:22bb

Disable Nvidia Driver on HOST

To prevent the host from loading the Nvidia driver, add the Nvidia modules to the kernel’s blocklist:

$ vim /etc/modprobe.d/blacklist.conf

Add the following lines:

blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_modeset

Update the initramfs and reboot:

$ update-initramfs -u
$ reboot

Configure VM

Install Virt-manager

In this article, we used virt-manager as VM manager, first step is install virt-manager, suppose Ubuntu will install all relative packages.

$ apt install virt-manager 

If you’re using SSH with X11 forwarding (e.g., ssh -X host) or MobaXTerm on Windows, Virt-manager will display the remote X window. If neither method works, consider installing a KDE desktop on the host and accessing it via the BMC remote console.

(Optional) Install KDE Plasma Desktop:

(option)
$ apt install kde-plasma-desktop

Create VM image

Virt-manager creates fixed-size VM images by default. If you prefer dynamic allocation, create the image manually:

$ qemu-img create -f qcow2 ubuntu2204.qcow2 200G

Add Nvidia device to VM

If the host is configured correctly, Virt-manager will list all PCIe devices, including the Nvidia GPU. Add the GPU and its audio device (e.g., 0005:01:00.0 and 0005:01:00.1) to the VM’s hardware list.

After adding the devices, proceed with the Ubuntu 22.04 installation.

After add hardware

Now, it can run begin install to install ubuntu 22.04

Disable secure Boot in UEFI

By default, Virt-manager enables Secure Boot. However, Nvidia drivers may not work with Secure Boot enabled. Even though the Nvidia installer includes a driver signing feature, the driver may still fail to load. To avoid issues, disable Secure Boot in the VM’s UEFI settings.

During the VM’s boot process, press the DEL key to enter UEFI settings and uncheck the Secure Boot option.

Before installing the Nvidia driver, ensure the necessary development packages are installed:

$ apt install build-essential

Then, install the Nvidia driver and reboot the VM.

GPU Passthrough Test

If everything is set up correctly, running nvidia-smi should display the GPU’s status.

For testing, you can use Ollama with the Deepseek-R1 model. Install Ollama with:

curl -fsSL https://ollama.com/install.sh | sh

Pull the Deepseek-R1 model. Since the GPU has 16GB of memory, the 14B model is a good choice (it requires ~10GB):

ollama run deepseek-r1:14b

Ask a question like, “Why is the sky blue?” This will trigger the model’s Chain-of-Thought (CoT) reasoning.

Monitor the GPU’s status using nvidia-smi to ensure it’s functioning correctly.

The post GPU Passthrough on ARM64 with Libvirt/Virt-manager appeared first on richliu's blog.

In this article, it will walk through the process of installiung Stable Diffusion WebUI on ARM64 + Nvidia RTX platform.
For such as hardware combinarion, it needs to solve some compatiability issues, so, here will describe how to install it and fix those issues.

Platform

CPU: Ampere Altra ARM64 CPU
Board: AsRock ALTRAD8UD
OS: Ubuntu 22.04

The platform is the most powerful ARM64 platform for a homelab that we can obtain today. There are some server models available, but they come with high-speed cooling fans that are not suitable for home use.

Nvidia has provided comprehensive library support for the ARM64 platform, including drivers, CUDA, and cuDNN. However, despite these efforts, there are still gaps in support. Even Nvidia’s Jetson product line lacks adequate support for ARM64 platforms in certain applications such as PyTorch. While there exists an individual PyTorch accelerator package for direct download on x64 platforms, the same cannot be said for ARM64 platforms. Although there is technically one package available, its availability is severely limited, rendering it practically useless.

The only viable solution to overcome these limitations is to utilize Nvidia’s Docker service to install Stable Diffusion.

Installation

Install Ubuntu 22.04 without selecting the HWE (Hardware Enablement) support. It appears that the current Nvidia driver version has issues, leading to potential soft-lock bugs. A general kernel version (5.15) is preferable for stability.

Next, install the necessary Ubuntu packages.

sudo apt-get install build-essential

Download Nvidia ARM64 (aarch64) Display Driver , current version is ok, when I write this article, current version is 535.154.05

Docker

Install Docker by following the Docker guide Install Docker Engine on Ubuntu,”. Also provide the necessary installation steps here. If already installed Docker or if these steps don’t work, please refer to the original website for updated instructions.

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Nvidia Container Toolkit

Follow the instructions for installing the NVIDIA Container Toolit. It can modify the Docker configuration using this toolkit. Alternatively, modifying the daemon.json file might achieve the same result, although I haven’t personally tested it. Below are the instructions:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

It can check the file /etc/docker/daemon.json file, the configuration should be like this.

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

Run Nvidia pytorch docker image

Following is example command to run the Nvidia pytorch docker image

sudo docker run --gpus all -it --rm \
        -v /host/work:/container/work \
        -p 7860:7860 \
        nvcr.io/nvidia/pytorch:XX.XX-py3

/host/work means work directory on host, /container/work means work directory in container(docker). -p 7860 mean gui default port, it needs to forward to host machine. XX.XX means container versiom please see tags to get container version, or this URL . [NGC Pytorch]. Following is example command.

sudo docker run --gpus all -it --rm \
        -v /work:/work \
        -p 7860:7860 \
        nvcr.io/nvidia/pytorch:24.01-py3

Stable Diffustion WebUI

First steps clone Stable Diffustion WebUI source code.

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

Then, it needs to modify the settings file. Edit webui.sh, and adjust the parameters to the following numbers.

use_venv=0
can_run_as_root=1

As we are operating within a container, there’s no need for a virtual environment (venv), and the script can be run as root. Modify the file webui-user.sh and add COMMANDLINE_ARGS accordingly.

export COMMANDLINE_ARGS="--no-half --precision full --listen"

or run it directly

export COMMANDLINE_ARGS="--no-half --precision full --listen" ; bash webui.sh

Currently, Stable Diffusion will download some packages, but errors may occur during the installation process. We’ll address and fix these errors in the next few steps.

CV2

Error message :

    if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 28, in __load_extra_py_code_for_module
    py_module = importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py", line 168, in <module>
    LayerId = cv2.dnn.DictValue
AttributeError: module 'cv2.dnn' has no attribute 'DictValue'

Modify file “/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py”, comment out line 169 like following.[ref]

# LayerId = cv2.dnn.DictValue

CodeFormer

Error message:

"/work/stable-diffusion-webui/repositories/CodeFormer/facelib/utils/face_restoration_helper.py", line 7, in <module>
        from facelib.detection import init_detection_model
      File "/work/stable-diffusion-webui/repositories/CodeFormer/facelib/detection/__init__.py", line 11, in <module>
        from .yolov5face.face_detector import YoloDetector
      File "/work/stable-diffusion-webui/repositories/CodeFormer/facelib/detection/yolov5face/face_detector.py", line 20, in <module>
        IS_HIGH_VERSION = tuple(map(int, torch.__version__.split('+')[0].split('.'))) >= (1, 9, 0)
    ValueError: invalid literal for int() with base 10: '0a0'

This issue caused by pytorch.version will return illegeal number like 2.2.0a0+81ea7a4, so it will fail to parse this mesage, temperory solution is to remove the version number, edit file “stable-diffusion-webui/repositories/CodeFormer/facelib/detection/yolov5face/face_detector.py”. and modify line 20, IS_HIGH_VERSION to following version to following:[ref]

version_string = torch.version.split('+')[0]
version_string = ''.join(filter(str.isdigit, version_string))
IS_HIGH_VERSION = tuple(map(int, version_string.split('.'))) >= (1, 9, 0)

Now, the Stable Diffusion GUI should be working as well as expected

The post How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform appeared first on richliu's blog.

Node: 'model_3/conv1d_24/Conv1D' 
DNN library is not found. 	 [[{{node model_3/conv1d_24/Conv1D}}]] [Op:__inference_train_function_34509]

It seems that there may be compatibility issues between the versions of TensorFlow/Keras installed outside of Conda and those installed within Conda. But it maybe also cause by tensorflow 2.12 has issue, please re-install 2.10 and try again .
Specified version can work well cuDNN 8.9, CUDA 11.8 and TensorFlow 2.10.
Here is how to re-install tensorflow 2.10 instruction.

pip uninstall tensorflow
pip install tensorflow-gpu
pip install tensorflow-gpu==2.10 

ref.
TensorFlow 2.12.0 WSL2 GPU support

Have tried someother way, all of them doesn’t work when you use 2.12 tensorflow. Don’t try this. ex:

sudo cp cuda/include/cudnn*.h   /anaconda3/envs/<your environment here>/include
sudo cp cuda/lib64/libcudnn*    /anaconda3/envs/<your environment here>/lib
sudo chmod a+r /usr/local/cuda/include/cudnn*.h    /anaconda3/envs/<your environment here>/lib/libcudnn*

ref.
How to install latest cuDNN to conda?

Install

CUDA

Suppose it already install WSL2 on your system, please follow the CUDA on WSL User Guide to install WSL, for Tensorflow 2.12, suggestion CUDA version is 11.8. Can be found here CUDA Toolkit 11.8 Downloads

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-wsl-ubuntu-11-8-local_11.8.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-11-8-local_11.8.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda

cuDNN

It still needs to install CUDNN, follow this instruction to download CUDNN 8.9.x, Installing cuDNN on Linux. Download cuDNN needs to register Nvidia account.
Download address : cuDNN Download, it has cuDNN for CUDA 12 and CUDA 11, download for CUDA 11.

Add Path to System Path

Put following seting into ~/.bashrc, thus it can update when get bash console

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Follow tensorflow install guide to install tensorflow.

Tensorflow step by step instructions for WSL2

Python Virtual Environment

if use virtual venv, it also can work, just need to install right version tensorflow. and it needs to configure cuDNN path on activate script

vim <path to venv>/bin/activate

# After get VIRTUAL_ENV, include the CUDNN path
export VIRTUAL_ENV

CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$VIRTUAL_ENV/lib/:$CUDNN_PATH/lib

Reference

WSL2 安裝 CUDA Toolkit、cuDNN

The post Run Keras or TensorFlow in WSL2 with Nvidia GPU appeared first on richliu's blog.

因為之前想要在 Windows 玩遊戲，又想要學一些 AI 的東西，所以後來裝了 WSL2 加上 Nvidia driver

如果要在這個環境下跑 FauxPilot ，那就要裝 Docker Windows 版，然後開啟 WSL2 的支援，雖然 Microsoft 也有說明，我覺得 Docker 這邊的說明比較好

現在 FauxPilot 的版本已經可以支援 8GB 的顯示卡了，選 7GB 的 codegen 就可以了
如果更大 codegen model 的可以用二張顯示卡，這樣就可以支援了

我照圖中的步驟安裝沒有什麼問題，有問題的應該是 VSCode copilot 的部份，按照 Fauxpilot 設定，並不會轉到 localhost:5000 ，猜測可能已經拿掉支援了或是我沒設定好

不過上面那個 openai 的 Sample code 就是可以直接存取，那個 sample code 不需要取得 openai 的 api key，直接用就可以，應該是 Nvidia server 自帶的功能

有人寫了一個 vscode-fauxpilot 的 extension ，下載位置在這邊，試了一下可以玩，簡單的測試了一下
要什麼功能就先 ### 你要的的功能，接下來就會顯示程式碼，再用 tab 就可以補完了，還頗有趣的

ref.
跑在本機的 GitHub Copilot 替代品

The post FauxPilot local Copilot Server appeared first on richliu's blog.

Tried to uninstall anydesk, nomachine and gcin, but those actions cannot solve this issue.

but one time, when plasma cpu usage 100%, the nvidia-smi command shows GPU usage 100%, so I think whether if prevent plasma to use nvidia, it might solve this issue temporary ?

Notebook has intel i915 driver, so, prevent plasma to use GPU can solve this issue.

First modify /usr/share/X11/xorg.conf.d/11-nvidia-prime.conf and replace it with text below

# DO NOT EDIT. AUTOMATICALLY GENERATED BY gpu-manager

Section "OutputClass"
    Identifier "Nvidia Prime"
    MatchDriver "nvidia-drm"
    Driver "nvidia"
    Option "AllowEmptyInitialConfiguration"
    Option "IgnoreDisplayDevices" "CRT"
    # Option "PrimaryGPU" "Yes"
    ModulePath "/x86_64-linux-gnu/nvidia/xorg"
EndSection


Section "OutputClass"
    Identifier "intel"
    MatchDriver "i915"
    Driver "modesetting"
    Option "PrimaryGPU" "yes"
EndSection

Protect the file will not be overwrite by gpu-manager

chattr +i /usr/share/X11/xorg.conf.d/11-nvidia-prime.conf

Reboot system and run nvidia-smi, plasma will not use nvidia GPU, so my plasma cpu usage 100% issue disappear.

this solution might not solve all plasma 100% cpu usage issue, but might be one possible solution.

Detail information which can refer to this url How to configure iGPU for xserver and nvidia GPU for CUDA work

The post KDE Plasma CPU High Usage After Install Nvidia Driver appeared first on richliu's blog.

有看過官網的文章 Firefox is slow – How to make it faster, 但是仍然無效.

最近心血來潮, 用 Google 找 Firefox Nvidia slow, 發現很多人都和我遇到相同的問題, 但是大多數都是在 Linux 平台, 少數是在 Windows 平台,

大家都提到關掉 Nvidia 3D 加速的功能可以讓 Firefox 恢復正常, 實際執行之後, Firefox 終於恢復正常速度了.

因為我用的是 Thinkpad T420s, 有個 Notebook Nvs Optimus 技術, 所以可能是使用內建顯卡時, 速度就會正常, 如果是用 Nvidia 顯示時, 速度就會變慢.
具體狀況我也是沒有查到, 如果知道的人還望通知.

下載新版的 Nvidia Driver 有改善，但是經過一天的測試, 認為預設設定還是該使用 Intel 內顯效能比較好, 不要再用 Nvidia 3D 加速.

此圖是 Nvidia Firefox 的設定

照著設就好了.

留此文供大家 Google.

Ref
Why is Firefox running super slow and IE runs great? New details 03/29

The post T420s Windows Firefox 速度奇慢的問題 appeared first on richliu's blog.