Kernel Internals for Debug

Advanced12 min read

Assumes: 6.1 QNX, 6.2 AAOS

TL;DR

Two kernels live on a Qualcomm cockpit SoC: Linux (GKI + vendor modules for AAOS) and QNX microkernel (packaged in an IFS). Both are loaded by the boot chain, configured by device tree or buildfile, and debuggable via different tools. GKI (Generic Kernel Image) is Google’s standardized Android kernel — the same binary across all devices, with vendor-specific hardware support delivered as loadable modules in vendor_boot.img. QNX IFS (Image Filesystem) bundles the microkernel, startup code, drivers, and applications into a single bootable image. Understanding how these kernels are structured, configured, and loaded is the prerequisite for every debug technique in Module 10.

You’ve seen kernel panics in logcat, QNX crash dumps in slog2, and mysterious unable to load module errors. This page gives you the architectural knowledge to interpret those messages and know which knob to turn. Not glamorous, but everything above depends on it.

Two Kernels, One SoC

The hypervisor (QHEE) on SA8155P/SA8295P runs multiple guest operating systems, each with its own kernel:

flowchart TB
    subgraph hw ["Hardware (SA8155P / SA8295P)"]
        CPU["Kryo CPU cores"]
    end

    subgraph hyp ["QHEE Hypervisor (EL2)"]
        VCPU_QNX["vCPU pool\nQNX guest"]
        VCPU_AAOS["vCPU pool\nAAOS guest"]
    end

    subgraph qnx_vm ["QNX VM (EL1)"]
        QNX_K["QNX microkernel\n(procnto-smp-instr)"]
        QNX_DRV["Resource managers\n(drivers)"]
        QNX_APP["QNX applications"]
    end

    subgraph aaos_vm ["AAOS VM (EL1)"]
        GKI["Linux GKI kernel\n(boot.img)"]
        VMOD["Vendor modules\n(vendor_boot.img)"]
        ANDROID["Android framework\n+ apps"]
    end

    CPU --> hyp
    VCPU_QNX --> qnx_vm
    VCPU_AAOS --> aaos_vm

Each kernel is loaded independently by the boot chain (Page 5.2): QHEE initializes VM isolation at EL2, then ABL loads boot.img (Linux GKI) for the AAOS guest and the QNX IFS for the host/safety domain. They share physical hardware through QHEE but have completely separate kernel images, configurations, and debug tools.

Think About It

You load a vendor kernel module on a GKI kernel. It works on one build but fails after a kernel update. Why?

Reveal Answer

GKI enforces KMI (Kernel Module Interface) stability — each .ko embeds a vermagic string that must exactly match the running kernel from uname -r. After a kernel update, boot.img changed but your modules in vendor_boot.img were built against the old GKI, causing “Invalid module format” or vermagic mismatch on load. Compare modinfo *.ko | grep vermagic against uname -r, then reflash both partitions from the same build or rebuild the module against matching GKI headers.

Linux GKI: The AAOS Kernel

What GKI Means

GKI stands for Generic Kernel Image — Google’s initiative to ship a single, standardized kernel binary for all Android devices. Before GKI, every OEM/SoC vendor maintained a heavily modified kernel fork. GKI splits the kernel into:

flowchart LR
    subgraph google ["Google-maintained"]
        GKI_BIN["GKI kernel binary\n(Image.lz4 in boot.img)"]
        GKI_CFG["gki_defconfig\nstandardized config"]
    end

    subgraph vendor ["Qualcomm / Tier-1 maintained"]
        VMOD_BIN["Vendor modules (.ko)\nin vendor_boot.img"]
        V_CFG["Vendor defconfig fragments\nboard-specific features"]
        V_DT["Device tree blob (.dtb)\nboard hardware description"]
    end

    subgraph running ["Running System"]
        KERNEL["Loaded GKI kernel"]
        MODS["Loaded vendor modules"]
        DEVTREE["Parsed device tree"]
    end

    GKI_BIN --> KERNEL
    VMOD_BIN --> MODS
    V_DT --> DEVTREE
    KERNEL -->|"probes DT, loads modules"| MODS

Component	Partition	Who Builds It	Contents
GKI kernel	boot	Google (AOSP)	Image.lz4 — the core kernel binary
Vendor modules	vendor_boot	Qualcomm + Tier-1	.ko files — UFS host, display DRM, audio codec, camera, custom drivers
Device tree blob	dtbo or embedded	BSP (Page 5.6)	.dtb — hardware description compiled from .dts
Vendor ramdisk	vendor_boot	Qualcomm + Tier-1	Init scripts, fstab, early-mount config

Memorization aid: GKI = Google’s Kernel Image. If Google didn’t build it, it’s a vendor module. The GKI kernel is the standardized foundation; vendor modules are the per-board customization layer.

How Modules Load

At boot, the GKI kernel parses the device tree, then loads vendor modules that match hardware nodes:

sequenceDiagram
    participant ABL as ABL Bootloader
    participant K as GKI Kernel
    participant DT as Device Tree
    participant MOD as Vendor Modules

    ABL->>K: Load Image.lz4 + DTB
    K->>DT: Parse hardware nodes
    K->>K: Mount vendor ramdisk
    loop For each DT node with status=okay
        K->>MOD: Find .ko matching compatible string
        MOD-->>K: Module loaded (or "module not found")
    end
    K->>K: Continue to Android init

When module loading fails, you see errors like:

[    2.841523] platform 1d84000.serial: deferred probe - no matching driver
[    3.102847] vendor_module: module xyz not found in vendor_boot

Module loading three-point check

When you see unable to load module or deferred probe:

1. Is the module in vendor_boot? — List contents: adb shell ls /vendor/lib/modules/
2. Does the device tree enable the hardware? — Check: adb shell cat /proc/device-tree/.../status
3. Is the kernel config set? — Verify: adb shell zcat /proc/config.gz | grep CONFIG_FEATURE

This three-point check resolves 80% of driver issues. The answer is almost always: module missing from image, DT node disabled, or config flag not set.

QNX IFS: The Self-Contained Image

QNX takes the opposite approach from Linux’s modular design. The IFS (Image Filesystem) bundles everything into a single bootable image:

flowchart TB
    subgraph ifs ["QNX IFS (single image)"]
        STARTUP["startup-sa8155p\nHW init, MMU, interrupts"]
        PROCNTO["procnto-smp-instr\nmicrokernel + process manager"]
        DRIVERS["Resource managers\nio-pkt, devnp-*, devc-*, io-audio"]
        APPS["Applications\nslog2info, dumper, pidin, custom apps"]
        SCRIPTS["Boot scripts\nstartup.sh, mount commands"]
    end

    subgraph linux_compare ["Linux equivalent (for comparison)"]
        INITRAMFS["initramfs / vendor_boot"]
        VMLINUZ["vmlinuz (kernel)"]
        MODULES["/vendor/lib/modules/*.ko"]
        ROOTFS["rootfs + Android framework"]
    end

QNX IFS Component	Linux Equivalent	Purpose
startup-sa8155p	Kernel early boot + device tree parsing	Board-specific hardware init before microkernel
procnto-smp-instr	vmlinuz (monolithic kernel)	Microkernel: scheduling, IPC, memory management
Resource managers (devc-*, io-pkt)	Kernel modules (.ko)	Drivers — but run in user space, not kernel space
slog2info, pidin, dumper	logcat, ps, coredumpctl	System utilities
startup.sh	init.rc	Boot script that starts services in order

Key difference: QNX drivers (resource managers) run in user space. A buggy driver crashes its own process, not the kernel. In Linux, a buggy in-kernel driver can panic the entire system. This architectural choice is why QNX achieves ASIL D — driver isolation is built into the kernel model.

Memorization aid: IFS = It’s Fully Self-contained. Everything the QNX system needs to boot is in one image. No separate modules, no ramdisk, no initramfs dance.

Building an IFS

The IFS is built with mkifs using a buildfile that specifies what goes in:

# Simplified QNX IFS buildfile example
[virtual=aarch64le,raw] .bootstrap = {
    startup-sa8155p -v
    PATH=/proc/boot procnto-smp-instr -v
}

[+script] .script = {
    # Start process manager
    procmgr_symlink ../../proc/boot/libc.so /usr/lib/ldqnx-64.so.2

    # Start serial driver
    devc-serqcom -e -F -b115200 -c 7372800 0x994000,0x20 52

    # Start slogger
    slogger2

    # Start network
    io-pkt-v6-hc -d devnp-emac ...

    # Start custom services
    waitfor /dev/ser1
    [+session] sh &
}

Unlike Linux where you configure a kernel, rebuild, and reflash separate partitions, QNX rebuilds the entire IFS image. A change to any driver or application means rebuilding and reflashing the IFS.

Mental Model

Think of the two kernels as two types of restaurants. Linux GKI is a franchise — the kitchen equipment (GKI kernel) is standardized across all locations, but each restaurant adds its own menu items (vendor modules) and local decor (device tree). You can swap dishes without rebuilding the kitchen. QNX IFS is a food truck — kitchen, menu, staff, and seating are all one self-contained unit. To change anything, you rebuild the whole truck. The food truck is simpler to audit (everything is right there), which is why it gets the safety certification (ASIL D).

GKI = franchise menu (standardized). Vendor modules = local specials (customized).

Kernel Config That Matters for Debug

These kernel configuration flags control which debug features are available at runtime. Knowing them lets you answer “why doesn’t this debug tool work?” in seconds:

Linux Kernel Debug Configs

Config Flag	What It Enables	When You Need It
CONFIG_KGDB	Kernel GNU Debugger — live kernel debugging via serial	Stepping through kernel code, inspecting variables at a crash
CONFIG_KGDB_SERIAL_CONSOLE	KGDB over serial port (UART)	Connecting GDB to kernel via debug UART
CONFIG_DEBUG_FS	/sys/kernel/debug virtual filesystem	Accessing regulator states, clock trees, GPIO configs at runtime
CONFIG_FTRACE	Function tracer — runtime kernel profiling	Tracing function calls, measuring latencies, profiling drivers
CONFIG_TRACE_MMIO_ACCESS	Log MMIO register reads/writes via ftrace	Debugging driver-hardware register interaction, finding NoC hangs
CONFIG_DEBUG_INFO	Debug symbols in vmlinux	Making crash stacks human-readable (function names vs hex addresses)
CONFIG_CONSOLE_POLL	Polled I/O for serial console	Required for KGDB serial connection
CONFIG_DYNAMIC_DEBUG	Per-module debug print control	Enabling verbose logs for specific drivers without rebuilding
CONFIG_STACKTRACE	Stack trace support	Getting meaningful backtraces in crash dumps

The debug build shortcut: Qualcomm’s DEBUG_BUILD=1 flag automatically enables most of these. If you’re on a debug BSP (Page 5.6), these configs are already set. On a performance BSP, you’ll need to verify each one.

Common Gotcha

If your crash stack shows only hex addresses with no function names, the kernel was built without debug info. CONFIG_DEBUG_INFO significantly increases kernel image size (often 5-10x for vmlinux). Production builds strip it to save flash space and boot time. You can still decode the stack offline using an unstripped vmlinux from the build server — but you need the exact matching build.

QNX Debug Equivalents

QNX debug features are controlled by startup options and build variants rather than a defconfig:

QNX Mechanism	Linux Equivalent	What It Does
procnto-smp-instr (the -instr variant)	CONFIG_FTRACE	Instrumented kernel — enables tracelogger kernel event capture
dumper process	CONFIG_COREDUMP	Runs as a daemon, captures core dumps when processes crash
-v flag on startup/procnto	CONFIG_DYNAMIC_DEBUG	Verbose boot output — shows hardware init details
slogger2	CONFIG_LOG_BUF_SHIFT (dmesg buffer)	System-wide log buffer that all processes can write to
qconn	CONFIG_KGDB	Debug agent — allows GDB and Momentics IDE to connect remotely

Kernel Modules vs Built-in: When It Matters

On Linux, drivers can be compiled built-in (part of the kernel binary) or as modules (.ko files loaded at runtime). The choice affects debug workflows:

Aspect	Built-in	Module (.ko)
Loaded when	Always, at kernel start	On demand or at boot via modprobe
Crashable	Crash = kernel panic	Crash = kernel panic (still runs in kernel space)
Updatable	Requires new boot.img flash	Can update vendor_boot.img only
Debug symbols	In main vmlinux	In separate .ko file — need add-symbol-file in GDB
Loading order	Implicit (compile order)	Explicit (module dependencies, modules.dep)

Common module-related failures and their fixes:

# "Unknown symbol" — module depends on another that isn't loaded yet
dmesg | grep "Unknown symbol"
# Fix: Check modules.dep, load dependency first

# "Module verification failed" — signature mismatch
dmesg | grep "module verification"
# Fix: Module built against different kernel version. Rebuild against current GKI.

# "deferred probe" — hardware not ready when module loaded
dmesg | grep "deferred"
# Fix: Usually resolves itself after dependencies load. If persistent, check DT status.

On QNX, the equivalent distinction is between drivers compiled into the IFS (always available) and drivers started by boot scripts (can fail to start). Since QNX drivers are user-space resource managers, a “module load failure” on QNX means a process failed to start — check pidin and slog2info rather than dmesg.

Device Tree at Runtime

The device tree isn’t just a boot-time artifact. On a running Linux system, you can inspect and verify it:

# The entire device tree is exposed as a filesystem
ls /proc/device-tree/
# Output: #address-cells  #size-cells  aliases  chosen  compatible
#         cpus  firmware  memory  model  pmu  psci  reserved-memory
#         soc  timer

# Check a specific peripheral's status
cat /proc/device-tree/soc/geniqup@9c0000/serial@994000/status
# Output: "okay" (enabled) or "disabled"

# Read the compatible string to know which driver should bind
cat /proc/device-tree/soc/geniqup@9c0000/serial@994000/compatible
# Output: "qcom,geni-debug-uart"

# Decompile the full running device tree for offline analysis
adb shell "dtc -I fs /proc/device-tree -O dts 2>/dev/null" > running_dt.dts

# Compare running DT against what you expected
diff expected.dts running_dt.dts

When this saves you: You flash a new BSP but a peripheral isn’t working. Instead of rebuilding and reflashing, inspect the running device tree to verify the node is present and enabled. If the node is missing, the device tree overlay didn’t apply. If the node is present but status = "disabled", the board DTS override is missing.

GKI version and module compatibility

GKI enforces a Kernel Module Interface (KMI) — a stable set of exported symbols that vendor modules link against. If a vendor module was built against GKI version X but the kernel is version Y, the module will fail to load with a “version magic” mismatch.

Check the running kernel version:

adb shell uname -r
# Output: 5.15.123-android14-11-g1234abcd

Check a module’s expected version:

modinfo /vendor/lib/modules/qcom_xxx.ko | grep vermagic
# Output: vermagic: 5.15.123-android14-11-g1234abcd SMP preempt mod_unload aarch64

If these don’t match exactly, the module won’t load. This happens most often after partial OTA updates or when mixing boot.img from one build with vendor_boot.img from another.

Connect the Dots

Page 5.2 (boot chain) showed the sequence that loads these kernels — ABL for Linux, QHEE for QNX. Page 5.6 (BSP) covered the device tree and defconfig that configure them. Pages 6.1 and 6.2 (QNX and AAOS) covered the operating systems running on top. This page connects the “what gets loaded” to the “how it’s structured” — the prerequisite for Page 10.4 (software debug toolkit) where you’ll use KGDB, ftrace, slog2, and the other tools these kernel configs enable.

Quick Reference

1. GKI = Google’s standardized kernel binary. Vendor-specific hardware support ships as loadable .ko modules in vendor_boot.
2. QNX IFS = self-contained bootable image with microkernel + drivers + apps. No separate modules.
3. CONFIG_KGDB, CONFIG_FTRACE, CONFIG_DEBUG_INFO are the three kernel configs that unlock most debug techniques. Check if they’re enabled before assuming a tool is broken.
4. Module loading failures follow a three-point pattern: module present? DT node enabled? Config flag set?
5. The running device tree is inspectable at /proc/device-tree/ — always verify before rebuilding.

Check Your Understanding

1. What is the key architectural difference between Linux GKI kernel modules and QNX drivers?

Applied

A. QNX drivers require device tree bindings; Linux modules do not use DT at allB. Linux modules can be updated independently; QNX drivers cannot be patchedC. Linux modules are faster because they run in kernel space with zero-copy IPCD. QNX drivers run in user space (crash kills process, not kernel); Linux in kernel

2. A vendor module fails to load with 'version magic' mismatch. What is the root cause?

Applied

A. The device tree does not have a matching compatible string for that deviceB. The module .ko file is corrupted on the vendor_boot UFS flash partitionC. Module compiled against a different GKI version — KMI vermagic mismatchD. CONFIG_MODULES is disabled in the GKI defconfig so modprobe is blocked

3. Which kernel configuration flag must be enabled to use KGDB for live kernel debugging over UART?

Recall

A. CONFIG_DEBUG_INFO onlyB. CONFIG_STACKTRACE and CONFIG_DYNAMIC_DEBUGC. CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE, and CONFIG_CONSOLE_POLLD. CONFIG_DEBUG_FS and CONFIG_FTRACE

4. In GKI architecture, which partition contains the standardized Google-built kernel binary vs vendor-specific .ko modules?

Recall

A. GKI kernel in vendor_boot.img; modules in boot.imgB. Both in the QNX IFSC. Both in system.imgD. GKI kernel (`Image.lz4`) in boot.img; vendor modules in vendor_boot.img

5. dmesg shows `platform 1d84000.serial: deferred probe - no matching driver`. Your three-point check order?

Applied

A. Reflash the QNX IFS image, then cold reboot the SoC and retry the testB. Replace the PMIC and check for CRANK brownout voltage dips on the benchC. Check VHAL gRPC vsock connection health between AAOS and QNX VMs firstD. Verify module in /vendor/lib/modules/, DT status=okay, CONFIG flag enabled

6. Post-OTA, AAOS boots but camera, audio, and display all fail with vermagic errors. boot.img is build A, vendor_boot.img is build B. Why doesn't QNX IFS reflashing fix this?

Synthesis

Connects to: Module 8 — Build, Flash & Update

A. QNX IFS write operations corrupt the AAOS vendor partition at shared offsetsB. The hypervisor disables kernel module loading after every OTA update cycleC. VHAL server blocks Linux module loading until all properties are registeredD. GKI modules live in vendor_boot.img, not QNX IFS — Linux partitions only

Module Review: Three Operating Systems

Question 1 of 12

Reading /proc/device-tree/chosen/stdout-path on AAOS shows serial@99c0000 disabled. UART works on QNX slog but not on AAOS dmesg console. Explanation?

🔗 Connects to: Module 5 — BSP

A. AAOS uses USB-only console and cannot be configured for serial UART outputB. UART requires a separate physical transceiver chip for each operating systemC. AAOS does not support UART at all — Linux removed serial console supportD. UART owned by QNX (slog2) — DT node disabled for AAOS; one VM per peripheral

Further Reading

• Linux Kernel Documentation — Official kernel docs covering modules, device tree, ftrace, KGDB, and debugfs — the AAOS GKI debug toolchain foundation.
• Android Generic Kernel Image (GKI) — How Google standardizes the Android kernel binary and ships vendor hardware support as loadable modules.
• QNX Neutrino Microkernel Architecture — Microkernel design, IFS structure, and user-space resource managers contrasted with Linux’s monolithic model.
• Building a QNX IFS Image (mkifs) — How QNX packages kernel, drivers, and apps into a single bootable image — the opposite of GKI’s modular approach.
• Linux Kernel ftrace Documentation — Function tracing and latency profiling on AAOS — requires CONFIG_FTRACE covered on this page.
• Linux Device Tree Usage — How the running device tree at /proc/device-tree/ describes hardware and drives module loading.
• Real-Time Scheduling (Linux) — SCHED_FIFO/RR and priority inheritance concepts — useful when comparing Linux guest scheduling to QNX deterministic IPC.