Launch QEMU with gdb and connect locally with gdb client¶
This guide is for cases where QEMU counters very early failures and it is hard to synchronize it in a later point in time.
Image creation and PVC population¶
This scenario is a slight variation of the guide about starting strace, hence some of the details on the image build and the PVC population are simply skipped and explained in the other section.
In this example, QEMU will be launched with gdbserver
and later we will connect to it using a local gdb
client.
The wrapping script looks like:
#!/bin/bash
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/run/debug/usr/lib64 /var/run/debug/usr/bin/gdbserver \
localhost:1234 \
/usr/libexec/qemu-kvm $@ &
printf "%d" $(pgrep gdbserver) > /run/libvirt/qemu/run/default_vmi-debug-tools.pid
First, we need to build and push the image with the wrapping script and the gdbserver:
FROM quay.io/centos/centos:stream9 as build
ENV DIR /debug-tools
ENV DEBUGINFOD_URLS https://debuginfod.centos.org/
RUN mkdir -p ${DIR}/logs
RUN yum install --installroot=${DIR} -y gdb-gdbserver && yum clean all
COPY ./wrap_qemu_gdb.sh $DIR/wrap_qemu_gdb.sh
RUN chmod 0755 ${DIR}/wrap_qemu_gdb.sh
RUN chown 107:107 ${DIR}/wrap_qemu_gdb.sh
RUN chown 107:107 ${DIR}/logs
Then, we can create and populate the debug-tools
PVC as with did in the strace example:
$ k apply -f debug-tools-pvc.yaml
persistentvolumeclaim/debug-tools created
$ kubectl apply -f populate-job-pvc.yaml
job.batch/populate-pvc created
$ $ kubectl get jobs
NAME COMPLETIONS DURATION AGE
populate-pvc 1/1 7s 2m12s
Configmap:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config-map
data:
my_script.sh: |
#!/bin/sh
tempFile=`mktemp --dry-run`
echo $4 > $tempFile
sed -i "s|<emulator>/usr/libexec/qemu-kvm</emulator>|<emulator>/var/run/debug/wrap_qemu_gdb.sh</emulator>|" $tempFile
cat $tempFile
As last step, we need to create the configmaps to modify the VM XML:
Build client image¶
In this scenario, we use an additional container image containing gdb
and the same qemu binary as the target process to debug. This image will be run locally with podman
.
In order to build this image, we need to identify the image of the virt-launcher
container we want to debug. Based on the KubeVirt installation, the namespace and the name of the KubeVirt CR could vary. In this example, we'll assume that KubeVirt CR is called kubevirt
and installed in the kubevirt
namespace.
You can easily find out the right names in your cluster by searching with:
The steps to build the image are:
-
Get the registry of the images of the KubeVirt installation:
-
Get the shasum of the virt-launcher image:
Example of Dockerfile:
ARG registry
ARG tag
FROM ${registry}/kubevirt/virt-launcher${tag} AS launcher
FROM quay.io/centos/centos:stream9 as build
RUN yum install -y gdb && yum clean all
COPY --from=launcher /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm
- Build the image by using the
registry
and thetag
retrieved in the previous steps:
Podman will replace the registry and tag arguments provided on the command line. In this way, we can specify the image registry and shasum for the KubeVirt version to debug.
Run the VM to troubleshoot¶
For this example, we add an annotation to keep the virt-launcher pod running even if any errors occur:
Then, we can launch the VM:
$ kubectl apply -f debug-vmi.yaml
virtualmachineinstance.kubevirt.io/vmi-debug-tools created
$ kubectl get vmi
NAME AGE PHASE IP NODENAME READY
vmi-debug-tools 28s Scheduled node01 False
$ kubectl get po
NAME READY STATUS RESTARTS AGE
populate-pvc-dnxld 0/1 Completed 0 4m17s
virt-launcher-vmi-debug-tools-tfh28 4/4 Running 0 25s
The wrapping script starts the gdbserver
and expose in the port 1234
inside the container. In order to be able to connect remotely to the gdbserver, we can use the command kubectl port-forward
to expose the gdb port on our machine.
$ kubectl port-forward virt-launcher-vmi-debug-tools-tfh28 1234
Forwarding from 127.0.0.1:1234 -> 1234
Forwarding from [::1]:1234 -> 1234
Finally, we can start the gbd client in the container:
$ podman run -ti --network host gdb-client:latest
$ gdb /usr/libexec/qemu-kvm -ex 'target remote localhost:1234'
GNU gdb (GDB) Red Hat Enterprise Linux 10.2-12.el9
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
--Type <RET> for more, q to quit, c to continue without paging--
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/libexec/qemu-kvm...
Reading symbols from /root/.cache/debuginfod_client/26221a84fabd219a68445ad0cc87283e881fda15/debuginfo...
Remote debugging using localhost:1234
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...
Downloading separate debug info for /system-supplied DSO at 0x7ffc10eff000...
0x00007f1a70225e70 in _start () from target:/lib64/ld-linux-x86-64.so.2
For simplicity, we started podman with the option --network host
in this way, the container is able to access any port mapped on the host.