A quick look inside the Android emulator

http://benno.id.au/blog/2007/11/29/android-qemu

 

I've been working on getting OKL4 up and running on theAndroid simulator. In doing so I've ended up getting quite involvedwith the simulator code base and thought I would share some of myfindings.

At the core the simulator is Qemu version 0.8.2. The Android teamhas provided a wrapper main(), which does some slightlynicer argument naming and parsing, and ends up calling the traditionalQemu entrypoint. Inside, the main change is that a new platform calledgoldfish has been added to supplement the existing Integratorand Versatile platforms.

When porting a different OS to a new platform, the first thing youneed to do is get some basic device drivers, such as interrupt controller,serial console and timer, up and running. Usually, the way to do thisis find the published spec sheet, and go off that. Unfortunately thereis no published spec sheet for the goldfish, but we have something infinitely more useful; the actual source code to the simulated device.(The number of times I've found bit errors in device documentation ispretty amazing!).

This post will share some of the details of the simulated platform(as it stands at this point in time!), along with some commentary. I'monly covering the bits that I have needed in bootstrapping OKL4, sokeypads, framebuffers, etc will have to wait for another day. (Hopefully,tomorrow).

Physical memory layout

The physical memory layout is about as simple as it gets. RAMstarts at address 0, and continues up to size of ram, in one contiguousblock.

Cache

The data cache is 16KiB, 4 way set associative, with 32 byte lines.This is pretty standard, although it would be nice to have higherassociativity. (Of course, it makes not very much difference in termsof simulation, so one can only guess that this cache layout is going to be similar to some real system-on-chip being used in an actual phone.)

Interrupt controller

The interrupt controller has a 4KiB block of registers residing at0xff000000. It consist of 5 32-bit registers.

STATUS at offset 0x0 contains thenumber of pending interrupt. It is a read-only register.

NUMBER at offset 0x4 containsthe lowest pending, enabled interrupt number. It is a read-onlyregister.

DISABLE_ALL at offset 0x8 isa write-only register. Writing any value to it will disableall interrupts.

DISABLE at offset 0xC is a write-only register. Writing an interrupt number to itwill disable to specified interrupt.

ENABLE at offset 0x10 is a write-only register. Writing an interrupt number to itwill enable to specified interrupt.

This has to be the best interface to aninterrupt controller ever. No messy shifting, or updating multipleregisters to get the job done. Every function I need to implementin my driver ends up being just a register read or write. Bliss!

Serial

The serial controller has a 4KiB block of registers residing at0xff002000. It consist of 5 32-bit registers.

PUT_CHAR at offset 0x0 is awrite-only register. Writing a value to it puts a characteronto the console.

BYTES_READY at offset 0x4 returnsthe number of characters waiting to be read from the console. This register is read-only.

CMD at offset 0x8 is a write-only register. Writing a command performs one of four actions.

• CMD_INT_DISABLE (0) disables the console interrupt.
• CMD_INT_ENABLE (1) enables the console interrupt.
• CMD_WRITE_BUFFER (2) copies DATA_LENbytes from virtual address DATA_PTR to the console.
• CMD_READ_BUFFER (3) copies DATA_LENbytes from the console to virtual address DATA_PTR. The numberof bytes should not exceed that specified by BYTES_READY.

DATA_PTR at offset 0x10 is a write-only register. The value in this register is thevirtual address used in read and write buffer commands.

DATA_LEN at offset 0x14 is a write-only register. The value in this register is thenumber of bytes to copy on the read or write buffer commands.

This is a really nice interface. My one reservation is that itwould be really nice if performing a read from PUT_CHARreturned a character if available. (Of course then it should be renamed fromPUT_CHAR.) It was an interesting decision to use virtual addresses forthe buffers, rather than a physical address. This will be different frommost hardware out there.

Timer

The serial controller has a 4KiB block of registers residing at0xff003000. It consist of 5 32-bit registers. Time isrepresented by a flowing 64-bit counter.

TIME_LOW at offset 0x0return the lowest 32-bit from the 64-bit counter. It also latchesthe high 32-bits into TIME_HIGH. You must readTIME_LOW, before reading TIME_HIGH to getconsistent values. It is a read-onlyregister.

TIME_HIGH at offset 0x4is a read-only register storing the top 32-bitsof the 64-bit counter. It should only be read after readingthe TIME_LOW value.

ALARM_LOW at offset 0x8 is awrite-only register storing the lowest 32-bits of thenext alarm value. When written it takes the top 32-bits forthe alarm value from ALARM_HIGH and stores the valuein an internal register. To get consistent results theALARM_HIGH should be stored first when setting analarm. When the counter value reaches the alarm value and interrupt istriggered.

ALARM_HIGH at offset 0xc isa write-only register storing the top 32-bitsof the next alarm value. Writing to this register does not updatethe internal 64-bit alarm register. This is done on writes to ALARM_LOW

CLEAR_INTERRUPT at offset 0x10is a write-only register. When written to itwill clear an interrupt previously posted by the alarm.

This is a nice simple way to access an OS timer. The only thingmissing is a periodic mode so the next alarm value doesn't need to becalculated each time. (Of course, periodic ticks are on the way out,so this isn't very critical.)

Bug fixes

The first bug fix is to actually make the thing compile on mymachine. This mostly involved removing what seems to be dead code. SDLis used, but the build system is set up to only use SDL on specificfiles, and on those files, the correct include would be #include<SDL.h>. It turns out this code is unused, so we can justget rid of it entirely. This isn't really a problem for anyone usingAndroid, just if you want to try and recompile.

--- android-emulator-20071111.orig/qemu/vl.c 2007-11-12 17:58:42.000000000 +1100
+++ android-emulator-20071111/qemu/vl.c   2007-11-29 00:29:35.000000000 +1100
@@ -78,12 +78,6 @@
 extern void  android_emulation_setup( void );
 extern void  android_emulation_teardown( void );
 
-#ifdef CONFIG_SDL
-#ifdef __APPLE__
-#include 
-#endif
-#endif /* CONFIG_SDL */
-
 #ifdef CONFIG_COCOA
 #undef main
 #define main qemu_main

The next bug is a little odd. This was found during a run of L4 test. Basically, the code goes to a lot of trouble to register a real-time alarm, and hook up a signalhandler to service this. (This is how timer interrupts end up being injected intothe emulated machine.) The problem is that the default sigprogmask seems to have SIGALRM blocked, which means we don't end up getting timer interrupts, or at leastnot if the emulated code is running in a tight loop. This bug could actually affect peopleusing Android. It is possible that in this case timer interrupts are missed and thesystem becomes unresponsive. I'm not sure if it is something strange in my setupthat makes SIGALRM blocked by default, or if this is a general problem. It shouldprobably be investigated further.

diff -ru android-emulator-20071111.orig/qemu/vl.c android-emulator-20071111/qemu/vl.c
--- android-emulator-20071111.orig/qemu/vl.c 2007-11-12 17:58:42.000000000 +1100
+++ android-emulator-20071111/qemu/vl.c   2007-11-29 00:43:34.000000000 +1100
@@ -1282,6 +1276,7 @@
     {
         struct sigaction act;
         struct itimerval itv;
+        sigset_t nset;
 
         sigfillset(&act.sa_mask);
         act.sa_flags = 0;
@@ -1304,6 +1299,10 @@
         /* we probe the tick duration of the kernel to inform the user if
            the emulated kernel requested a too high timer frequency */
         getitimer(ITIMER_REAL, &itv);
+
+        sigemptyset(&nset);
+        sigaddset(&nset, SIGALRM);
+        sigprocmask(SIG_UNBLOCK, &nset, NULL);
     }
 #endif
 }

The next bugs are nasty. Really nasty. And I didn't really debugthem, I kind of guessed and looked at diffs to find them. They dealwith the innards of the ARM MMU, and are only really exposed bykernels that make full use of the memory management unit (e.g: domainsfor fast context switching, super pages, PID relocation). OKL4does this, which I why I'm hitting these bugs, where as Linux doesn't right now, so it avoids them. The first of these problems was fixedupstream in Qemu 0.9.0, the second was actually found by anotherengineer at OK Labs, Matt Warton, andhas been pushed upstream already.

diff -ru android-emulator-20071111.orig/qemu/target-arm/helper.c android-emulator-20071111/qemu/target-arm/helper.c
--- android-emulator-20071111.orig/qemu/target-arm/helper.c 2007-11-12 17:58:42.000000000 +1100
+++ android-emulator-20071111/qemu/target-arm/helper.c   2007-11-29 00:26:44.000000000 +1100
@@ -247,7 +247,7 @@
 
   switch (ap) {
   case 0:
-      if (access_type != 1)
+      if (access_type == 1)
           return 0;
       switch ((env->cp15.c1_sys >> 8) & 3) {
       case 1:
@@ -428,6 +428,7 @@
         break;
     case 3: /* MMU Domain access control.  */
         env->cp15.c3 = val;
+        tlb_flush(env, 1);
         break;
     case 4: /* Reserved.  */
         goto bad_reg;

New features

Since I was having to modify the emulator anyway, I decided to adda feature to make my live just a little easier. Qemu expects to be loadingand a Linux kernel, but not all kernels out there are Linux, and they havedifferent expectations about where they should be loaded and what data ifany should be passed to them.

I've extended Qemu, and the android wrapper to support a new-os-type flag, so that you can specify what type of OS isbeing emulated. By default this is set to Linux, and the normal Linuxkernel loading algorithm applies. If it is set to anything else, thenit simply loads the specified kernel directly in at the start of memoryand doesn't do any string or command line passing.

diff -ru android-emulator-20071111.orig/qemu/android_sdl.c android-emulator-20071111/qemu/android_sdl.c
--- android-emulator-20071111.orig/qemu/android_sdl.c 2007-11-12 17:58:41.000000000 +1100
+++ android-emulator-20071111/qemu/android_sdl.c   2007-11-29 00:59:24.000000000 +1100
@@ -3537,6 +3537,7 @@
 static char *arg_nand0 = 0;
 static char *arg_nand1 = 0;
 static char *arg_sdcard = 0;
+static char *arg_os_type = 0;
 static char *arg_kernel = 0;
 static char *arg_ramdisk = 0;
 static char *arg_tracefile = 0;
@@ -3573,6 +3574,7 @@
     const char *help;   /* description text for this option */
 } argmap[] = {
     { "-system",   &arg_sysdir,    0,              0,                   "<dir>",    "search system, ramdisk and userdata images in <dir>" },
+    { "-os-type",  &arg_os_type,   0,             "linux",              "<os-type>","kernel image is of given OS type. E.g: linux, okl4" },
     { "-kernel",   &arg_kernel,    0,             "kernel-qemu",        "<file>",   "use <file> as the emulated kernel" },
     { "-ramdisk",  &arg_ramdisk,   0,             "ramdisk.img",        "<file>",   "use <file> as the ramdisk image (default is <system>/ramdisk.img)" },
     { "-image",    &arg_nand0,     0,             "system.img",         "<file>",   "use <file> as the system image (default is <system>/system.img)" },
@@ -4270,6 +4272,10 @@
 
     n = 1;
     /* generate arguments for the underlying qemu main() */
+    if(arg_os_type && arg_os_type[0]) {
+        args[n++] = "-os-type";
+        args[n++] = arg_os_type;
+    }
     if(arg_kernel && arg_kernel[0]) {
         args[n++] = "-kernel";
         args[n++] = arg_kernel;
--- android-emulator-20071111.orig/qemu/vl.c 2007-11-12 17:58:42.000000000 +1100
+++ android-emulator-20071111/qemu/vl.c   2007-11-29 00:56:47.000000000 +1100
@@ -199,6 +193,8 @@
 int dcache_store_miss_penalty = 5;
 #endif
 
+char *os_type = "linux";
+
 extern void  dprint( const char* format, ... );
 
 /***********************************************************/
@@ -6005,6 +6006,7 @@
     QEMU_OPTION_smb,
     QEMU_OPTION_redir,
 
+    QEMU_OPTION_os_type,
     QEMU_OPTION_kernel,
     QEMU_OPTION_append,
     QEMU_OPTION_initrd,
@@ -6095,6 +6097,7 @@
     { "redir", HAS_ARG, QEMU_OPTION_redir },
 #endif
 
+    { "os-type", HAS_ARG, QEMU_OPTION_os_type },
     { "kernel", HAS_ARG, QEMU_OPTION_kernel },
     { "append", HAS_ARG, QEMU_OPTION_append },
     { "initrd", HAS_ARG, QEMU_OPTION_initrd },
@@ -6564,6 +6567,9 @@
                 pstrcpy(serial_devices[0], sizeof(serial_devices[0]), "stdio");
                 nographic = 1;
                 break;
+            case QEMU_OPTION_os_type:
+                os_type = optarg;
+                break;
             case QEMU_OPTION_kernel:
                 kernel_filename = optarg;
                 break;
--- android-emulator-20071111.orig/qemu/hw/arm_boot.c 2007-11-12 17:58:41.000000000 +1100
+++ android-emulator-20071111/qemu/hw/arm_boot.c   2007-11-29 00:57:04.000000000 +1100
@@ -64,6 +64,8 @@
     stl_raw(p++, 0);
 }
 
+extern char *os_type;
+
 void arm_load_kernel(int ram_size, const char *kernel_filename,
                      const char *kernel_cmdline, const char *initrd_filename,
                      int board_id)
@@ -71,19 +73,27 @@
     int kernel_size;
     int initrd_size;
     int n;
+    int linux_image = (strcmp(os_type, "linux") == 0);
 
     /* Load the kernel.  */
     if (!kernel_filename) {
         fprintf(stderr, "Kernel image must be specified/n");
         exit(1);
     }
-    kernel_size = load_image(kernel_filename,
-                             phys_ram_base + KERNEL_LOAD_ADDR);
+
+    if (linux_image) {
+      kernel_size = load_image(kernel_filename,
+               phys_ram_base + KERNEL_LOAD_ADDR);
+    } else {
+      kernel_size = load_image(kernel_filename,
+               phys_ram_base);
+    }
     if (kernel_size < 0) {
         fprintf(stderr, "qemu: could not load kernel '%s'/n", kernel_filename);
         exit(1);
     }
-    if (initrd_filename) {
+
+    if (linux_image && initrd_filename) {
         initrd_size = load_image(initrd_filename,
                                  phys_ram_base + INITRD_LOAD_ADDR);
         if (initrd_size < 0) {
@@ -94,12 +104,14 @@
     } else {
         initrd_size = 0;
     }
-    bootloader[1] |= board_id & 0xff;
-    bootloader[2] |= (board_id >> 8) & 0xff;
-    bootloader[5] = KERNEL_ARGS_ADDR;
-    bootloader[6] = KERNEL_LOAD_ADDR;
-    for (n = 0; n < sizeof(bootloader) / 4; n++)
-        stl_raw(phys_ram_base + (n * 4), bootloader[n]);
-    set_kernel_args(ram_size, initrd_size, kernel_cmdline);
+    if (linux_image) {
+        bootloader[1] |= board_id & 0xff;
+        bootloader[2] |= (board_id >> 8) & 0xff;
+        bootloader[5] = KERNEL_ARGS_ADDR;
+        bootloader[6] = KERNEL_LOAD_ADDR;
+        for (n = 0; n < sizeof(bootloader) / 4; n++)
+            stl_raw(phys_ram_base + (n * 4), bootloader[n]);
+        set_kernel_args(ram_size, initrd_size, kernel_cmdline);
+    }
 }