Running Mac OS X as a QEMU/KVM Guest

Gabriel L. Somlo

Last updated: Fri. Mar. 21, 2014
Feedback to: somlo at cmu dot edu

0. I Just Want It Working, Right Now !

OK, here's what you'll need (or skip to the technical details instead): Once the above components are in place, you'll need a HDD image for your Mac OS X guest:
	cd /home/$(whoami)/OSXGUEST
	qemu-img create -f qcow2 mac_hdd.img 30G
To start your Mac OS X guest in QEMU, use the following command line:
	bin/qemu-system-x86_64 -enable-kvm -m 2048 -cpu core2duo -M q35 \
	  -usb -device usb-kbd -device usb-mouse \
	  -device isa-applesmc,osk="insert-real-64-byte-OSK-string-here" \
	  -bios bios-mac.bin -kernel ./chameleon_svn2360_boot \
	  -device ide-drive,bus=ide.2,drive=MacHDD \
	  -drive id=MacHDD,if=none,file=./mac_10.8.img \
	  -monitor stdio
When running the OS X guest for the first time, you'll need to install the operating system to the HDD image, so you'll need to add (and boot from) an install DVD image on the command line.
	  -device ide-drive,bus=ide.0,drive=MacDVD \
	  -drive id=MacDVD,if=none,snapshot=on,file=./MountainLion.10.8.5.iso
SnowLeopard (10.6) was the last OS X version released in DVD format. Starting with Lion (10.7), an install DVD (.iso) image may be generated on an OS X machine by following the example in
this script. The example demonstrates the procedure for Mavericks, but both Lion and MountainLion work the exact same way.

Another optional command line parameter could be used to start an SMP guest, such as:
	  -smp 4,cores=2
There are still a few remaining issues to be worked out: Legal Disclaimer: I am not a lawyer, and this is not legal advice. Taking into account that OS X is now officially supported on commercial virtualization solutions such as VMWare Fusion and Parallels, and after a careful reading of Apple's OS X EULA (which states that "[...] you are granted a [...] license to install, use and run one (1) copy of the Apple Software on a single Apple-Branded computer at any one time"), it is my belief that it's OK to run Mac OS X as a QEMU guest, provided the host hardware is a genuine, Apple-manufactured Mac computer, running an arbitrary (e.g. Linux) host OS. This happens to be how I'm using it, but YMMV.

Please read on for a detailed overview of all the issues involved !

1. Intro, Motivation, and Overview

This work started out as my OS Practicum project during the Fall of 2012, where I picked Mac OS X on QEMU/KVM as my topic. I'm a "native" Linux user, but my IT department supports quite a few Mac OS X users, and I decided I could be much faster and more productive if I could access a Mac as a VM guest rather than dual-boot or cold-start an additional machine each time it was needed.

The original work to enable OS X to run under QEMU/KVM was done by Alexander Graf. Additional kernel and userspace patches were contributed by René Rebe. Previously available patches and documentation I reviewed include: Several different components must successfully interact with each other to facilitate running Mac OS X as a QEMU/KVM guest. Each component is listed below, with a link to its dedicated section covering relevant aspects and outstanding issues related to Mac OS X guest support:

2. KVM Kernel Module

KVM is the component located at the lowest level, providing an interface to the virtualization support offered by (among a few others) the x86_64 hardware architecture. Let's begin with a brief list of upstream resources: Earlier OS X versions (up to 10.7 a.k.a. Lion) indiscriminately use MONITOR and MWAIT instructions, which are not currently supported by KVM and which result in an "invalid opcode" fault when a guest attempts to execute them. A thorough explanation is given here. Beginning with 10.8 (MountainLion), the OS X kernel follows the best practice of checking CPUID for MONITOR/MWAIT support, and will not attempt to use them if not supported. To run 10.7 or earlier versions of OS X, either apply this additional KVM patch, or, on an already installed HDD image, run:
	rm -rf System/Library/Extensions/AppleIntelCPUPowerManagement.kext
to prevent OS X from attempting to use the unsupported instructions.

Currently, there is one remaining KVM patch required to boot OS X. This is a misleadingly simple-looking one liner, which removes a single line from virt/kvm/ioapic.c (or x86/ioapic.c in the kvm-kmod version):
-	irq_level ^= entry.fields.polarity;
Without removing this line, a Mac OS X guest hangs during boot with an error message that reads: "still waiting for root device".

Modern x86 system design includes Intel's IOAPIC (interrupt controller) chip, which receives interrupts from peripheral devices and forwards them to the appropriate CPU's local interrupt controller (LAPIC). The IOAPIC can be programmed to handle both level and edge triggered interrupts. Historically, with the IOAPIC's predecessor (the 8259 PIC), level triggered interrupts were considered "active" (or "asserted") when the line went "high", and "inactive" when the line remained "low", from an electrical standpoint. In other words, the "logical" state of an interrupt line coincided with its "physical" state. With the IOAPIC, it is possible to configure each individual input (interrupt line) to use the "classic" interpretation of "active" vs. "inactive", now referred to as "ActiveHigh", or to reverse it by setting a per-input-line "polarity bit", so that interrupts are considered asserted when the line drops to "low" from a default inactive "high" voltage state. This mode is known as "ActiveLow". With "ActiveLow", the physical state of the line is the exact opposite of its logical state.

ACPI-compliant operating systems are expected to query the firmware for an indication of which polarity type (ActiveLow or ActiveHigh) to use for any devices with level-triggered interrupts, and to configure the IOAPIC registers accordingly. Both QEMU and KVM have accumulated a significant number of optimizations based on the assumption that guest operating systems use ActiveHigh polarity, and are coded to assume that "physical" and "logical" IRQ line states are in sync. Even when a misbehaving guest OS (you guessed it, OS X does this) ignores the ACPI polarity hint (which in QEMU/KVM is ActiveLow, i.e. "physical"=="logical") and configures the virtual IOAPIC with the wrong polarity bit values, both QEMU and KVM wil mostly use "logical" IRQ line levels.

The referenced patch removes the last place where KVM attempted to care about the value of the polarity bit placed into the IOAPIC register by the guest OS. Because logical levels were assumed everywhere else, looking at the polarity bit could result in flipping the given "logical" line state at the wrong time, resulting in level triggered interrupts being ignored instead of handled. Once we stop paying attention to the polarity bit values, things work fine. Logical IRQ levels are passed into KVM from userspace by QEMU, and whatever the guest OS does when it programs its IOAPIC polarity bits doesn't matter. Thus, when OS X wrongfully ignores QEMU's ACPI and proceeds with a hard-coded ActiveLow polarity, things just work in spite of that.

3. QEMU

QEMU is a multi-architecture emulator running in user-space. For certain architectures (such as x86_64), QEMU is able to take advantage of hardware virtualization features offered through e.g. KVM, which allow it to run guests at near-native performance levels. Here is roughly how QEMU and KVM work together to implement a guest VM: A quick list of QEMU upstream resources might include: There are four relevant topics regarding support for Mac OS X guests: QEMU's recent inclusion of support for the Q35/ICH9 based architecture; emulation of Apple's System Management Controller (a.k.a. AppleSMC), including automatically advertising its presence via ACPI; and, finally, a few issues related to QEMU's emulation of the e1000 network controller.

3.1. The Q35/ICH9 architecture

Support for the Q35 architecture was recently (Dec. 2012) merged into QEMU mainline. Q35 replaces the old I440FX (a.k.a. PIIX) with Intel's more modern ICH9 chipset, which also happens to be used on most Intel-based Apple hardware. Among other hardware, ICH9 includes an integrated AHCI disk controller, which had to be added explicitly prior to the Q35 QEMU command line:
	bin/qemu-system-x86_64 -enable-kvm -m 2048 -cpu core2duo \
	  -usb -device usb-kbd -device usb-mouse \
	  -device isa-applesmc,osk="insert-real-64-byte-OSK-string-here" \
	  -bios bios-mac.bin -kernel ./chameleon_svn2360_boot \
	  -device ahci,id=ide \
	  -device ide-drive,bus=ide.2,drive=MacHDD \
	  -drive id=MacHDD,if=none,file=./mac_10.8.img \
	  -monitor stdio
As Q35 is slated to become the new default "machine type" in QEMU in the near future, the bulk of the effort (development, debugging, and testing) to get Mac OS X supported under QEMU will be focused on this platform.

3.2. The AppleSMC emulator

The AppleSMC (or System Management Controller) is a chip specific to Intel-based computers manufactured by Apple. Its main purpose is to control (and report on) fan speeds, temperature sensors, screen and keyboard light intensity levels, and miscellaneous other power management features.

From the point of view of the operating system driver, interaction with the SMC happens via port-based I/O: The name of a 4-character key is written to a control port, and the key's numeric or ASCII value is then read from (or written to) a data port. Keys typically represent fan speeds, light intensity levels, or temperatures.

A pair of already upstreamed patches advertises the presence of the SMC in QEMU's ACPI/DSDT table, conditionally, based on whether or not it was specified on the QEMU command line.

There are currently two outstanding issues with QEMU's AppleSMC emulation which could improve Mac OS X guest support, outlined below.
3.2.1. Automatic OSK "pass-through" on Apple hardware
The AppleSMC is also used to store a 64-byte ASCII string copyrighted by Apple, spread across two 32-byte key values, named OSK0 and OSK1. This string is used by Mac OS X to determine whether it's being booted on genuine Apple hardware. QEMU does not set up AppleSMC emulation by default (since only OS X guests require it at this time). To set it up, the following QEMU command line snippet is required:
	-device isa-applesmc,osk="insert-real-64-byte-OSK-string-here"
The user must supply the correct value of the 64-byte OSK string as an argument, and is responsible for honoring Apple's OS X EULA (which states that "[...] you are granted a [...] license to instal, use and run one (1) copy of the Apple Software on a single Apple-Branded computer at any one time").

I wrote a small C program, SmcDumpHw.c, which can be used to read various SMC key values (including OSK0 and OSK1) from an Intel Mac running Linux. However, a significant improvement in usability and ease of compliance with the OS X EULA could be accomplished by allowing QEMU's AppleSMC emulator to automatically acquire the OSK strings from the underlying (Apple) host hardware.

Currently, the drivers/hwmon/applesmc.c Linux driver populates a Sysfs directory (/sys/devices/platform/applesmc.768/) which offers access to most SMC key values. Unfortunately, that does not include OSK0 and OSK1. I submitted this patch against the applesmc.c Linux driver, but encountered two main objections (also see the various other replies in the referenced thread): I'm planning to follow up with the Linux maintainer of applesmc.c again in the near future to reiterate these points, but in the mean time I could really use some advice and feedback from anyone with successful patch submission experience to the Linux kernel :)
3.2.2. OS X fails to write a key value to the emulated SMC
During boot, Mac OS X 10.6 (Snow Leopard, the only version currently working under QEMU) logs a few non-fatal SMC-related errors:
	SMC::smcGetVersWithSMC ERROR: smcReadKey REV failed (0xff)
	SMC::smcInitHelper ERROR: smcPublishVersion failed (0xff)
	SMC::smcInitHelper ERROR: smcPublishShutdownCause failed (0xff)
	SMC::smcPublisVersion ERROR: smcGetVers for SMC 0 failed (0xff)
	SMC::smcNotificationPublishedHandler ERROR: smcWriteKey NTOK failed (0xff), will not receive interrupts
It appears that emulating a few extra SMC keys, as well as allowing the OS X guest to write (as opposed to just read) some of the supported keys might make these errors go away.

3.3. The e1000 virtual network card

By default, QEMU uses the emulated e1000 network controller when starting a new guest, unless a different model is explicitly requested. Older versions of QEMU defaulted to the rtl8139 controller. OS X 10.6 (Snow Leopard) can use either model, but as a QEMU guest was unable to recognize the e1000 on older versions of QEMU, and is unable to recognize the rtl8139 on current QEMU git master. Since the e1000 is a more modern controller (as well as the new default), I decided not to spend any time figuring out why the rtl8139 isn't recognized, and get e1000 to work properly instead.

An already committed patch fixes the failure of OS X to configure the default MAC address (reporting "00:00:00:00:00:00" instead). OS X most likely expects the Apple EFI BIOS to initialize the network controller to the point where its capable of listening for packets targeted at its own factory-default MAC address. Most other operating systems will take care of all this from within their respective drivers, but for now OS X needs QEMU to take care of this part on its behalf.

Even once OS X can detect the e1000 network card with the correct MAC address, we are left with a failure to negotiate a virtual Ethernet "link" once the guest completes its boot sequence. My QEMU e1000 patch fixes that problem, but whether it's acceptable for upstream or not is currently under debate. The alternative would be to teach either Chameleon or SeaBIOS enough about the e1000 card to have one of them perform the initialization steps that Apple's EFI BIOS (or bootloader) performs on behalf of OS X "in the wild". If this turns out to be workable, the fix described in the above paragraph (QEMU commit 372254c6e5c078fb13b236bb648d2b9b2b0c70f1) should also be reverted, and its functionality added to Chameleon or SeaBIOS, respectively.

Finally, we have the issue of Mavericks (OS X 10.9) being unable to configure (i.e., load the driver for) the default PCI e1000 network card (E1000_DEVID = E1000_DEV_ID_82540EM) exposed by QEMU with the default shipping driver (AppleIntel8254XEthernet.kext), and that of the working alternative (E1000_DEVID = E1000_DEV_ID_82545EM_COPPER) not working on Windows.

Currently, there is a patch by Romain Dolbeau winding its way through the QEMU review process, which would allow us to pick the specific e1000 "flavor" best suited for each individual guest.

3.4. Boot OS X on QEMU without KVM hardware assistance

This part isn't truly necessary for supporting production OS X guests expected to do real work. It would however be interesting for debugging purposes to be able to run OS X in fully emulated mode, without any KVM hardware assistance. The latter (ioapic) patch is already upstream, and harmonizes QEMU's emulated ioapic behavior with KVM.

4. SeaBIOS

SeaBIOS is the default BIOS for QEMU/KVM. The QEMU source tree includes a relatively up-to-date snapshot of SeaBIOS (in pre-built, binary form). However, supporting Mac OS X currently requires patching SeaBIOS, and therefore we build a separate binary (bios-mac.bin) from current git master, and use it to start the OS X guest instead. Upstream resources include: Among other things, SeaBIOS provides the guest with a SMBIOS (a.k.a. DMI) table, containing a list of "hardware" components, along with versioning and revision information. OS X expects the memory device entries to comply with version 2.3 of the spec, as well as the presence of a Type 2 (Baseboard) entry.

Lion (10.7) and MountainLion (10.8) crash during boot in the absence of the Type 2 (Baseboard) SMBIOS entry. All versions (10.6 through 10.9) will have the GUI crash and restart when "About This Mac" is selected, unless the Type 17 (Memory Device) SMBIOS entries comply with at least v2.3 of the spec. The submitted patches may not make it upstream but you may use this cumulative patch instead, for now. In the long run, I plan on adding proper SMBIOS table generation functionality to QEMU, which will supersede the tables generated in SeaBIOS itself. Once that's in place, we will no longer have to patch SeaBIOS at all, and will be able to simply use the default blob shipping with QEMU.

5. Chameleon

Chameleon is a Darwin/XNU boot loader based on Apple's boot-132. It is currently used to boot OS X on systems which do not contain Apple's EFI BIOS as expected. Project resources include: Chameleon boots the OS X kernel (XNU) in three stages. We are interested mainly in the last stage (known as "stage 2", counting from 0). The project is designed to be built under OS X itself, and also depends on Xcode. Xcode v5.0.2 (free/beer download via the AppStore) on OS X 10.8 or later can build the latest Chameleon trunk (SVN 2360 at the time of this writing). Once these prerequisites are met, simply running 'make' in the appropriate SVN subdirectory will build the entire project.

The Chameleon stage-2 bootloader will be built as "./sym/i386/boot". On QEMU, we bypass all earlier stages by loading this binary as the default "kernel" (via the "-kernel chameleon_svn2360_boot" command line argument). On a (U)EFI compliant machine, in theory, (U)EFI would find a file named "/System/Library/CoreServices/boot.efi" on the OS X partition and use it directly. Chameleon's stage-2 "boot" file is a roughly equivalent replacement for that.

Chameleon offers a range of additional "bells and whistles", such as the ability to load alternative and/or additional .kext modules (OS X kernel drivers), and replace the system's default ACPI DSDT, mostly intended for "hackintosh" systems. On QEMU, however, the idea is to offer a DSDT and virtualized "hardware" that's close enough for OS X to run on, without needing that type of "assistance" from the bootloader.

In the long term, it might be possible to use a QEMU-compatible UEFI BIOS instead of SeaBIOS, and have it boot into OS X using the original /System/Library/CoreServices/boot.efi Apple-provided bootloader. Alternatively, it would be nice to port or rewrite the relevant subset of Chameleon's stage-2 bootloader to build on Linux. We should be able to leave out all DSDT and .kext-related parts, as they will not needed with QEMU.

6. Conclusion and Future Work

My ultimate goal is to upstream whatever changes are needed to KVM/QEMU/SeaBIOS/Chameleon/etc., to allow installing and running OS X from unadulterated, standard images. I plan to keep this document updated with the current list of outstanding issues. Eventually, the idea is for this document to "melt away" as all these issues are addressed, and all patches are accepted into their respective upstream projects.

UPDATE: I am Alex Graf's "assistant mentor" for
this OSX on QEMU related GSOC-2014 project. If you're a student, and are interested, check it out!

I appreciate any ideas, comments, and feedback. Please email me at somlo at cmu dot edu with any questions, comments, and suggestions.