Introduction: How to debug code using Corellium
Testing Corellium for another release we found out that Xcode is unable to prepare some iOS 15.x virtual devices for development. Moreover, Xcode doesn't return any errors and seems to be just stuck during the process. It was a little bit odd because we had this feature functioning for some iOS 15.4 betas and everything was working fine. Further investigation revealed that the problem is limited to particular devices (iPhone 11/12/13) and only appears on iOS 15.4 beta 2 and later.
One of the most useful features of Corellium is the ability to choose any iOS version starting with 10.3. Therefore we can easily debug and trace iOS 15.4 beta 1 and beta 2 side by side looking for differences until we spot the error.
This post is a walkthrough of our usual process in order to find and solve various bugs in Corellium.
Where to start with iOS debugging: The Developer Disk Image (DDI)
In order to prepare device for development, Xcode uploads and mounts a special disk image to the iOS device which includes debugserver and other files. You can see its contents by looking into /Developer folder on iOS filesystem:
iPhone:~ root# ls -al /Developer
total 8
drwxr-xr-x 7 root wheel 306 Feb 23 01:48 ./
drwxr-xr-x 26 root wheel 832 Apr 23 19:49 ../
-rw-r--r-- 1 root wheel 4348 Feb 23 01:48 .TrustCache
drwxrwxr-x 3 root admin 102 Feb 23 01:48 Applications/
drwxrwxr-x 9 root admin 306 Feb 23 01:48 Library/
drwxr-xr-x 3 root wheel 102 Feb 23 01:48 System/
drwxrwxr-x 6 root admin 204 Feb 23 01:48 usr/
Also when it is successfully mounted, we usually see the following lines in kernel console log:
hfs: mounted DeveloperDiskImage on device disk2
AMFI: Successfully loaded a new image4 v1 trust cache with 78 entries.
During tests we can see this log for iOS 15.4 beta 1 but not for beta 2. So it is pretty clear that beta 2 is unable to mount the image for some reason and we need to find out why.
Reading Syslog like a map
Reading syslog is the rule of thumb to deal with various issues on iOS. However, by default some important information in the log is hidden under the <private> output, which exists to limit sensitive information from being disclosed in the log during normal operations. This can be disabled by adding a com.apple.system.logging.plist file into the /Library/Preferences/Logging/ folder on iOS filesystem.
The plist file must contain this XML document:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Enable-Logging</key>
<true/>
<key>Enable-Private-Data</key>
<true/>
</dict>
</plist>
Now we can capture syslog for both versions and compare them side by side.
DDI mounting process
Grepping logs with various words like "mount" or "diskimage", we were able to define binaries involved in the mounting process.
These are the the key events:
lockdownd: spawn_xpc_service: service com.apple.mobile.storage_mounter_proxy started
...
MobileStorageMounter: new xpc connection: mobile_storage_proxy<283>
...
MobileStorageMounter: (DiskImages2) [com.apple.DiskImages2.Default] -[DIBaseXPCHandler createConnection]: Creating connection with com.apple.diskimagescontroller
...
diskimagescontroller: [com.apple.DiskImages2.Default] -[DIBaseAgentXPCHandler createConnection]: Creating connection to com.apple.diskimagesiod.xpc
These events form the following sequence:
lockdownd → mobile\_storage\_proxy → MobileStorageMounter → diskimagescontroller → diskimagesiod
We are not going to dig into every single one of these binaries and instead we try to find the last matching log entry. Fortunately it is very easy to spot by filtering by the modules above:
Mounting process never reaches following code in diskimagesiod (circled above with yellow):
-[DIDeviceHandle updateBSDNameWithBlockDevice:error:]: BSD name: disk2
This means it is time to jump into IDA Pro.
Digging Into diskimageiod
Searching by BSD name: we can find following string %.*s: BSD name: %{public}@ referenced in -[DIDeviceHandle updateBSDNameWithBlockDevice:error:] function. As we know this code is not called for beta 2 so we need to walk backwards to determine the callers and the overall sequence of events.
The caller is -[DIDeviceHandle waitForDeviceWithError:] but importantly, updateBSDNameWithBlockDevice is called only when some flag is set:
port = IONotificationPortCreate(kIOMainPortDefault);
if ( port ) {
RunLoopSource = IONotificationPortGetRunLoopSource(port);
if ( RunLoopSource ) {
CFRunLoopAddSource(v10, RunLoopSource, kCFRunLoopDefaultMode);
serv_match = IOServiceMatching("IOMedia");
if ( !IOServiceAddMatchingNotification(port, "IOServiceMatched", serv_match, IOMediaMatchingCallback, &refCon, ¬ification) ) {
IOMediaMatchingCallback((__int64)&refCon, notification);
while ( !flag ) {
... // STUCK HERE ?
}
IOObjectRelease(notification);
}
CFRunLoopRemoveSource(v10, v15, kCFRunLoopDefaultMode);
}
else {
...
}
IONotificationPortDestroy(v11);
}
if ( flag ) {
v20 = -[DIDeviceHandle updateBSDNameWithBlockDevice:error:](v7, a3);
flag = (char)v20;
}
And the flag is set in IOMediaMatchingCallback when there is an IOMedia service match:
diIOIterator = +[DIIOIterator initWithIOIterator:retryIfInvalidated:retain:](notification, 1LL, 1LL);
diIOObject = +[DIIOObject initWithIteratorNext:](diIOIterator);
if ( diIOObject ) {
diIOObject_tmp1 = 0LL;
do {
iterator = -[diIOObject newIteratorWithOptions:retryIfInvalidated:error:](3LL, 1LL, 0LL);
if ( iterator ) {
while ( 1 ) {
diIOObject_tmp2 = +[DIIOObject initWithIteratorNext:](iterator);
objc_release(diIOObject_tmp1);
if ( !diIOObject_tmp2 ) {
diIOObject_tmp1 = 0LL;
goto inner_bail;
}
ioObj = -[diIOObject_tmp2 ioObj];
if ( IOObjectConformsTo(ioObj, "IOMedia") )
break;
if ( IOObjectIsEqualTo(ioObj_, *(_DWORD *)(arg + 8)) ) {
**(_BYTE **)arg = 1; // FLAG SET !
...
}
...
Looking into all these DIIO... classes we found out that these are just a simple wrappers around IOKit functions like IOIterator, etc.
-[DIDeviceHandle waitForDeviceWithError:] is indeed a very interesting function because there is also a while loop which spins until flag is set. Can it be stuck there maybe? Let's check this out with lldb:
(lldb) process connect connect://localhost:4919
(lldb) process attach --name diskimagesiod --waitfor
Process 604 stopped
* thread #1, stop reason = signal SIGSTOP
frame #0: 0x0000000102fa227c dyld`dyld3::MachOFile::isDylib() const
dyld`dyld3::MachOFile::isDylib:
-> 0x102fa227c <+0>: ldr w8, [x0, #0xc]
0x102fa2280 <+4>: cmp w8, #0x6 ; =0x6
0x102fa2284 <+8>: cset w0, eq
0x102fa2288 <+12>: ret
Target 0: (diskimagesiod) stopped.
Executable module set to "/usr/libexec/diskimagesiod".
(lldb) continue
Process 604 resuming
(lldb) process interrupt
Process 604 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
frame #0: 0x00000001b7e58500 libsystem_kernel.dylib`mach_msg_trap + 8
libsystem_kernel.dylib`mach_msg_trap:
-> 0x1b7e58500 <+8>: ret
libsystem_kernel.dylib`mach_msg_overwrite_trap:
0x1b7e58504 <+0>: mov x16, #-0x20
0x1b7e58508 <+4>: svc #0x80
0x1b7e5850c <+8>: ret
Target 0: (diskimagesiod) stopped.
(lldb) bt
* thread #2
* frame #0: 0x00000001b7e58500 libsystem_kernel.dylib`mach_msg_trap + 8
...
frame #6: 0x0000000102d89624 diskimagesiod`___lldb_unnamed_symbol12$$diskimagesiod + 452
...
frame #15: 0x00000001f1a79104 libsystem_pthread.dylib`_pthread_wqthread + 228
(lldb) frame select 6
frame #6: 0x0000000102d89624 diskimagesiod`___lldb_unnamed_symbol12$$diskimagesiod + 452
diskimagesiod`___lldb_unnamed_symbol12$$diskimagesiod:
-> 0x102d89624 <+452>: mov x0, x25
0x102d89628 <+456>: bl 0x102e50328 ; symbol stub for: objc_release
0x102d8962c <+460>: ldrb w8, [sp, #0x2f]
0x102d89630 <+464>: cbz w8, 0x102d895f0 ; <+400>
This perfectly matches the loop code:
__text:0000000100005624 MOV X0, X25 ; id
__text:0000000100005628 BL _objc_release
__text:000000010000562C LDRB W8, [SP,#0x80+flag]
__text:0000000100005630 CBZ W8, loc_1000055F0
Now we know for sure that diskimageiod is stuck in a loop because the IOMedia service for DDI is not found in IORegistry. Fortunately, there is an ioreg tool to check if it is actually there:
iPhone:~ root# ioreg | grep IOMedia
| | | | +-o APPLE SSD AP0016H Media <class IOMedia, id 0x1000004f2, registered, matched, active, busy 0 (183 ms), retain 12>
| | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x10000053b, registered, matched, active, busy 0 (6 ms), retain 6>
| | | | +-o Container@1 <class IOMedia, id 0x100000599, registered, matched, active, busy 0 (52 ms), retain 12>
| | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x10000059b, registered, matched, active, busy 0 (19 ms), retain 7>
| | | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005ad, registered, matched, active, busy 0 (11 ms), retain 7>
| | | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005b2, registered, matched, active, busy 0 (11 ms), retain 7>
| | | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005b4, registered, matched, active, busy 0 (12 ms), retain 7>
| | | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005b3, registered, matched, active, busy 0 (10 ms), retain 7>
| | | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005b1, registered, matched, active, busy 0 (11 ms), retain 7>
| | | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005ae, registered, matched, active, busy 0 (11 ms), retain 7>
| | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005b0, registered, matched, active, busy 0 (11 ms), retain 7>
| | | | +-o APPLE SSD AP0016H Media <class IOMedia, id 0x1000005c3, registered, matched, active, busy 0 (5 ms), retain 10>
| | | | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000005c4, registered, matched, active, busy 0 (0 ms), retain 6>
| | +-o Apple Disk Image Media <class IOMedia, id 0x1000008da, registered, matched, active, busy 0 (8 ms), retain 9>
| | +-o IOMediaBSDClient <class IOMediaBSDClient, id 0x1000008db, registered, matched, active, busy 0 (1 ms), retain 6>
What a surprise. There is a Apple Disk Image Media <class IOMedia, ...> entry but it is ignored by IOIterator. The question is - can it be diskimageiod-specific or actually a system-wide problem?
DDI check tool
There is a relatively quick way to check for IOService match behaviour using IOKit framework. So we wrote a tool for that:
#include <stdio.h>
#include <IOKit/IOKitLib.h>
void DeviceAdded (void* refCon, io_iterator_t iterator) {
io_service_t service = 0;
printf("[+] L0: i = %x\n", iterator);
while ((service = IOIteratorNext(iterator)) != 0) {
io_name_t name;
uint64_t eid;
IORegistryEntryGetName(service, name);
IORegistryEntryGetRegistryEntryID(service, &eid);
printf("[+] L1: s = %.8llX %s\n", eid, name);
io_iterator_t iter;
if (!IORegistryEntryCreateIterator(service, "IOService", 3u, &iter)) {
printf("[+] L2: i = %x\n", iter);
io_service_t serv = 0;
while ((serv = IOIteratorNext(iter)) != 0) {
IORegistryEntryGetName(serv, name);
IORegistryEntryGetRegistryEntryID(serv, &eid);
if ( IOObjectConformsTo(serv, "IOMedia") ) {
printf("[+] L2: s = %.8llX %s (conforms to IOMedia)\n", eid, name);
IOObjectRelease(serv);
break;
}
printf("[+] L2: s = %.8llX %s\n", eid, name);
IOObjectRelease(serv);
}
}
}
}
int main (int argc, const char * argv[]) {
CFDictionaryRef matchingDict = NULL;
io_iterator_t iter = 0;
IONotificationPortRef notificationPort = NULL;
CFRunLoopSourceRef runLoopSource;
kern_return_t kr;
matchingDict = IOServiceMatching("IOMedia");
notificationPort = IONotificationPortCreate(kIOMasterPortDefault);
runLoopSource = IONotificationPortGetRunLoopSource(notificationPort);
CFRunLoopAddSource(CFRunLoopGetCurrent(), runLoopSource, kCFRunLoopDefaultMode);
kr = IOServiceAddMatchingNotification(notificationPort, kIOFirstMatchNotification, matchingDict, DeviceAdded, NULL, &iter);
DeviceAdded(NULL, iter);
printf("[+] --- wait for events\n");
CFRunLoopRun();
IONotificationPortDestroy(notificationPort);
IOObjectRelease(iter);
return 0;
}
Lets try it on beta 1 first (some output is removed for convenience):
iPhone:/var/mobile/Media root# ./ddicheck
[+] L0: i = e03
[+] L1: s = 1000005CC APPLE SSD AP0016H Media
[+] L2: i = 2503
[+] L2: s = 1000005CB IOBlockStorageDriver
...
[+] L2: s = 100000100 Root
[+] L1: s = 1000005C3 Update
[+] L2: i = 2403
[+] L2: s = 1000005B9 AppleAPFSContainer
[+] L2: s = 1000005B7 Container (conforms to IOMedia)
[+] L1: s = 1000005C2 Preboot
[+] L2: i = 1a03
[+] L2: s = 1000005B9 AppleAPFSContainer
[+] L2: s = 1000005B7 Container (conforms to IOMedia)
[+] L1: s = 1000005C1 Hardware
[+] L2: i = 2203
[+] L2: s = 1000005B9 AppleAPFSContainer
[+] L2: s = 1000005B7 Container (conforms to IOMedia)
[+] L1: s = 1000005BD Baseband Data
[+] L2: i = 1c03
[+] L2: s = 1000005B9 AppleAPFSContainer
[+] L2: s = 1000005B7 Container (conforms to IOMedia)
[+] L1: s = 1000005BC xART
[+] L2: i = 2003
[+] L2: s = 1000005B9 AppleAPFSContainer
[+] L2: s = 1000005B7 Container (conforms to IOMedia)
[+] L1: s = 1000005BB Data
[+] L2: i = 1e03
[+] L2: s = 1000005B9 AppleAPFSContainer
[+] L2: s = 1000005B7 Container (conforms to IOMedia)
[+] L1: s = 1000005BA System
[+] L2: i = 2a03
[+] L2: s = 1000005B9 AppleAPFSContainer
[+] L2: s = 1000005B7 Container (conforms to IOMedia)
[+] L1: s = 1000005B7 Container
[+] L2: i = 5403
[+] L2: s = 1000005B0 IOGUIDPartitionScheme
[+] L2: s = 1000005AC APPLE SSD AP0016H Media (conforms to IOMedia)
[+] L1: s = 1000005AC APPLE SSD AP0016H Media
[+] L2: i = 5303
[+] L2: s = 1000005A9 IOBlockStorageDriver
...
[+] L2: s = 100000100 Root
[+] --- wait for events
[+] L0: i = e03
[+] L1: s = 1000008FA Apple Disk Image Media
[+] L2: i = 5103
[+] L2: s = 1000008F9 IOBlockStorageDriver
...
[+] L2: s = 100000100 Root
And we can clearly see that 1000008FA Apple Disk Image Media shows up in the output when the DDI is mounted. However, running same test on beta 2 doesn't give us anything after --- wait for events during DDI mount.
This means it is a system-wide problem which is leading us to the XNU IOService implementation in the iOS kernelcache.
Tracing IOService::doServiceMatch()
In short the service matching process looks like this in XNU:
void IOService::doServiceMatch(...)
OSArray *IOService::copyNotifiers(...)
bool IOService::matchPassive(...)
bool IOService::matchInternal(...)
So let's find this code in the kernelcache. It is very easy to do using following marker:
void _IOConfigThread::main(void * arg, wait_result_t result) {
...
if (KERN_SUCCESS != kr) {
IOLog("thread_policy_set(%d)\n", kr);
}
...
case kMatchNubJob:
nub->doServiceMatch( job->options );
break;
...
}
Looking for the "thread_policy_set(%d)\n" string in kernelcache we can find it referenced inside of IOConfigThread::main and then going further down through calls matching the code flow with XNU source we can recover all four functions from above.
Let's describe the problem once again: Usermode listeners are not getting triggered because copyNotifiers() doesn't return the appropriate notifier for them. In turn, copyNotifiers() doesn't add the notifier to the array if matchPassive() returns false which happens if matchInternal() doesn't find matches in the provided dictionary.
Therefore, the plan is to trace service matching for beta 1 and beta 2 while running our ddicheck tool (to avoid a race with diskimageiod and guarantee at least one notifier) and then compare the trace output. In our case we need to log the bool result of the matchPassive() call for the IOMedia entry which is wrapped in the following object:
OSDictionary
{
OSDictionary::dictEntry
[
{ "IOProviderClass": "IOMedia" },
...
]
}
OSDictionary, OSString and other structures are defined in XNU source, so we know the offsets and can parse these on the fly.
OSDictionary {
+0x20: OSDictionary::dictEntry* [
{
+0x0: OSSymbol* { // key
+0x10: const char*
}
+0x8: OSObject* { // value
+0x10: const char*
}
},
...
]
}
There is one more trick worth mentioning before we start: XNU has a gIOKitDebug mask to extend IOKit-related logging. It is not difficult to spot it in matchPassive() function for example.
__TEXT_EXEC:__text:FFFFFFF0081E4440 ADRP X8, #qword_FFFFFFF009E9C168@PAGE
__TEXT_EXEC:__text:FFFFFFF0081E4444 LDR X0, [X8,#qword_FFFFFFF009E9C168@PAGEOFF]
__TEXT_EXEC:__text:FFFFFFF0081E4448 TBZ W0, #4, loc_FFFFFFF0081E4478
Setting it to 0xFFFFFFFF actually makes debugging a bit easier. Most likely it is controlled by some boot argument but for now let's attach IDA to the iOS kernel and the set this gIOKitDebug flag in memory.
If we put a breakpoint on the first instruction of IOService::doServiceMatch(...) and trigger the DDI mount from Xcode, then when it stops X16/X17 will hold the name of the current IOService object and it is there only because of the gIOKitDebug flag.
X16 FFFFFFE4CD118660 (MEMORY) -> ("bAppleDiskImageDevice")
X17 FFFFFFE4CD118660 (MEMORY) -> ("bAppleDiskImageDevice")
Debugging this code we defined four different threads calling IOService::doServiceMatch(...) for the following objects:
- AppleDiskImageDevice
- Apple Disk Image Media
- IOBlockStorageDriver
- IOMediaBSDClient
However, only one thread is calling IOService::invokeNotifier(...) function, and it is Apple Disk Image Media.
Now let's talk about tracing.
Unfortunately, we can't use Frida to perform hooks in the kernel. However, there is something else we can use instead - GDB monitor patch feature only available on Corellium via Hypervisor Hooks.
It works the following way: you specify kernel address and a short C-like snippet of code to be executed when the program counter hits the desired address.
Let's use the address followed by the IOService_matchPassive call for the hook:
__TEXT_EXEC:__text:FFFFFFF0081E323C LDR X1, [X25,#0x18] ; table
__TEXT_EXEC:__text:FFFFFFF0081E3240 MOV X0, X19 ; this
__TEXT_EXEC:__text:FFFFFFF0081E3244 MOV W2, #0 ; options
__TEXT_EXEC:__text:FFFFFFF0081E3248 BL IOService_matchPassive
__TEXT_EXEC:__text:FFFFFFF0081E324C CBZ W0, loc_FFFFFFF0081E3214 <--- HOOK
And finally craft the command to extract OSString from OSDictionary entry, compare it with the "IOMedia\0" (0x00616964654d4f49 in little endian) and print function result with the thread info:
patch 0xFFFFFFF0081E324C
u64 table = (*(u64 *)(cpu.x[25] + 0x18));
u64 entries = (*(u64 *)(table + 0x20));
u64 value = (*(u64 *)(entries + 0x08));
u64 strref = (*(u64 *)(value + 0x10));
u64 str = (*(u64 *)(strref));
if ( str == 0x00616964654d4f49ul ) {
print("[+] Match");
print_str(0, (void*)strref, 10);
print_int("", cpu.w[0]);
print_thread(0, 0);
print("\n");
}
This command is meant to be used as a one-liner, but we discovered that IDA Pro also handles it fine when you copy/paste this multiline text to IDA Pro's GDB field.
Having everything set for beta 1 we enable a breakpoint at the start of IOService::doServiceMatch(...) and trigger the DDI mount from Xcode one more time while running ddicheck tool.
When we see Apple Disk Image Media string in the X16 register, we need to capture the thread info which can be done by using the monitor thread GDB command:
GDB>thread
CPU 2: thread: 0xffffffe3e57f0000, tid: 11060, pid: 0, name: kernel_task
It is important to remember the thread ID (TID) in order to identify hooks related to IOMedia matches later.
Then resume execution and filter purple entries with tid: 11060 in the kernel console log:
[+] Match "IOMedia" : 1 thread: 0xffffffe3e57f0000, tid: 11060, pid: 0, name: kernel_task
[+] Match "IOMedia" : 0 thread: 0xffffffe3e57f0000, tid: 11060, pid: 0, name: kernel_task
[+] Match "IOMedia" : 0 thread: 0xffffffe3e57f0000, tid: 11060, pid: 0, name: kernel_task
[+] Match "IOMedia" : 1 thread: 0xffffffe3e57f0000, tid: 11060, pid: 0, name: kernel_task
Now let's perform same thing on beta 2 (hook address - 0xFFFFFFF008166AD8) and observe kernel console:
As you can see the last entry for tid 5806 has 0 returned from the IOService_matchPassive instead of 1 which appears on beta1.
It would have been very convenient if we could stop execution on a hook to debug further. And fortunately on Corellium we can! Lets modify our hook code a bit:
patch 0xFFFFFFF008166AD4
u64 table = (*(u64 *)(cpu.x[25] + 0x18));
u64 entries = (*(u64 *)(table + 0x20));
u64 value = (*(u64 *)(entries + 0x08));
u64 strref = (*(u64 *)(value + 0x10));
u64 str = (*(u64 *)(strref));
if ( str == 0x00616964654d4f49ul ) {
print("[+] Match");
print_str(0, (void*)strref, 10);
print_thread(0, 0);
print("\n");
debug();
}
Here we use address before the IOService_matchPassive call for the hook and trigger the debug() command which stops execution on IOMedia match. Resuming execution in IDA Pro for each hook we are looking for the fourth call for Apple Disk Image Media thread.
Then we can step through the code, especially the IOService::matchInternal function which basically returns due to the TBNZ W8, #0x1F, loc_FFFFFFF0081685F4 condition here:
__TEXT_EXEC:__text:FFFFFFF008167FDC LDR W8, [X21,#0x48]
__TEXT_EXEC:__text:FFFFFFF008167FE0 MVN W9, W8
__TEXT_EXEC:__text:FFFFFFF008167FE4 LSR W24, W9, #0x1F
__TEXT_EXEC:__text:FFFFFFF008167FE8 MOV W27, #1
__TEXT_EXEC:__text:FFFFFFF008167FEC TBNZ W8, #0x1F, loc_FFFFFFF0081685F4
X21 holds the this pointer for IOService*, and offset +0x48 points to its __state[0] member. This perfectly corresponds to the XNU source:
bool IOService::matchInternal(OSDictionary * table, uint32_t options, uint32_t * did) {
do{
...
match = (0 == (kIOServiceUserInvisibleMatchState & __state[0]));
if ((!match) || (done == count)) {
break;
}
...
}
...
return match;
}
And the state is 0x8000000E, which corresponds to kIOServiceUserInvisibleMatchState | kIOServiceFirstPublishState | kIOServiceMatchedState | kIOServiceRegisteredState.
Now we know that usermode clients are not getting notified because the IOMedia object is hidden in beta 2 and we just need to find out why.
Capture the gIOServiceHideIOMedia Flag
According to the XNU source, the kIOServiceUserInvisibleMatchState flag is set in IOService::doServiceMatch(...):
if (gIOServiceHideIOMedia && metaCast(gIOMediaKey) && !(kIOServiceUserUnhidden & __state[1])) {
if (gIOServiceRootMediaParent && !hasParent(gIOServiceRootMediaParent)) {
OSObject * prop = copyProperty(gPhysicalInterconnectKey, gIOServicePlane);
if (!prop || !prop->isEqualTo(gVirtualInterfaceKey)) {
__state[0] |= kIOServiceUserInvisibleMatchState;
}
OSSafeReleaseNULL(prop);
}
}
Let's debug this part on beta 1 to see why it is not reached there. Apparently it depends on another flag causing the following branch at 0xFFFFFFF0081E9628 to be taken:
__TEXT_EXEC:__text:FFFFFFF008167FDC LDR W8, [X21,#0x48]
__TEXT_EXEC:__text:FFFFFFF0081E9620 ADRP X8, #byte_FFFFFFF009EF1E4C@PAGE
__TEXT_EXEC:__text:FFFFFFF0081E9624 LDRB W8, [X8,#byte_FFFFFFF009EF1E4C@PAGEOFF]
__TEXT_EXEC:__text:FFFFFFF0081E9628 TBNZ W8, #0, loc_FFFFFFF0081E974C
Looks like byte_FFFFFFF009EF1E4C is the gIOServiceHideIOMedia flag. But hold on! if should be skipped if this flag is zero, not vice versa as it is in the decompiler.
if ( (byte_FFFFFFF009EF1E4C & 1) == 0 ) {
...
}
Let's check how it is set in the XNU source:
wasHiding = gIOServiceHideIOMedia;
if (wasHiding && !parent) {
gIOServiceHideIOMedia = false;
}
And in the kernelcache:
v7 = byte_FFFFFFF009EF1E4C;
if ( (byte_FFFFFFF009EF1E4C & 1) == 0 && !a1 )
byte_FFFFFFF009EF1E4C = 1;
But the weirdest part is that the beta 2 kernelcache XNU version is xnu-8020.100.406 and all sources before and after this version have the flag inverted.
In any case, if we put a breakpoint on that code we are able to recover the backtrace which corresponds to the following code flow:
void IOService::doServiceMatch(...)
UInt32 IOService::_adjustBusy(...)
void IOService::publishHiddenMedia(...)
Needless to say that IOService::publishHiddenMedia is not called on beta 2. Therefore we need to take a closer look at the IOService::_adjustBusy function.
XNU to the rescue (again)
Lets strip this function a bit for better understanding:
UInt32 IOService::_adjustBusy( SInt32 delta ) {
if (delta) {
do {
...
count = next->__state[1] & kIOServiceBusyStateMask;
wasQuiet = (0 == count);
...
count += delta;
next->__state[1] = (next->__state[1] & ~kIOServiceBusyStateMask) | count;
nowQuiet = (0 == count);
...
if ((wasQuiet || nowQuiet)) {
...
if (nowQuiet && (next == gIOServiceRoot)) {
if (gIOServiceHideIOMedia) {
publishHiddenMedia(NULL);
}
...
}
}
} while ((wasQuiet || nowQuiet) && (next = next->getProvider()));
}
}
Being called from the IOService::doServiceMatch with delta = -1 is our case, this function is trying to decrement a busy counter for each parent (provider) in the IORegistry branch from the current IOService up to the root (gIOServiceRoot). If all the nodes along the way are in quiet (non-busy) state, this essentially means there are no busy objects in the entire tree, and it is safe to publish IOMedia to usermode by calling publishHiddenMedia(NULL), if it hasn't been done yet.
At the same time, if there is one device which stays busy for some reason, IOMedia will never be published and we can use the ioreg tool to confirm that.
Sorry, I am busy
Let's restart both devices and wait a bit to let them finish booting, then check to see if there is anything busy in IORegistry.
Nothing found for beta 1:
iPhone:~ root# ioreg | grep "busy [1-9]"
iPhone:~ root#
And even after waiting for a while, there are couple of entries for beta 2:
iPhone:~ root# ioreg | grep "busy [1-9]"
+-o D53gAP <class IOPlatformExpertDevice, id 0x100000239, registered, matched, active, busy 1 (216482 ms), retain 33>
+-o AppleARMPE <class AppleARMPE, id 0x10000023c, registered, matched, active, busy 1 (216459 ms), retain 31>
| +-o arm-io@10F00000 <class IOPlatformDevice, id 0x10000011d, registered, matched, active, busy 1 (216528 ms), retain 119>
| | +-o AppleT810xIO <class AppleT810xIO, id 0x100000266, registered, matched, active, busy 3 (216476 ms), retain 121>
| | +-o aop@4A400000 <class AppleARMIODevice, id 0x10000012f, registered, matched, active, busy 1 (216474 ms), retain 9>
| | | +-o AppleASCWrapV4 <class AppleASCWrapV4, id 0x100000280, !registered, !matched, active, busy 1 (215853 ms), retain 5>
| | | +-o iop-aop-nub <class AppleA7IOPNub, id 0x100000130, registered, matched, active, busy 1 (215858 ms), retain 21>
| | | +-o RTBuddyV2 <class RTBuddyV2, id 0x100000475, registered, matched, active, busy 1 (215746 ms), retain 40>
| | | | +-o AOPEndpoint16 <class RTBuddyEndpointService, id 0x1000004ad, registered, matched, active, busy 1 (215742 ms), retain 7>
| | | | | +-o AppleSPU@1000001b <class AppleSPU, id 0x1000004c1, registered, matched, active, busy 1 (215743 ms), retain 8>
| | | | | +-o aop-audio <class AppleSPUAppInterface, id 0x10000013a, registered, matched, active, busy 1 (215720 ms), retain 22>
| | | | | +-o AppleAOPAudioController <class AppleAOPAudioController, id 0x10000054c, registered, matched, active, busy 2 (215674 ms), retain 29>
| | | | | +-o audio-haptic <class AppleAOPAudioDeviceNode, id 0x100000141, registered, !matched, active, busy 1 (215677 ms), retain 11>
| | | | | +-o audio-hpdbg <class AppleAOPAudioDeviceNode, id 0x100000142, registered, !matched, active, busy 1 (215678 ms), retain 11>
| | +-o alc0 <class AppleARMIODevice, id 0x1000001cf, registered, !matched, active, busy 1 (216680 ms), retain 10>
| | +-o alc2 <class AppleARMIODevice, id 0x1000001d1, registered, !matched, active, busy 1 (216681 ms), retain 10>
These are all audio related hardware: alc0, alc2, audio-haptic, audio-hpdbg.
The next step is to remove these from the iOS DeviceTree and see if it makes a difference.
Cutting device tree
There are several projects to deal with iOS DeviceTree binaries, but in Corellium we have our own tools to adjust the device tree on the fly for testing and engineering purpose.
In order to check the theory, we temporarily disabled compatible and device_type members for the objects above to prevent this hardware from being initialized.
device-tree/arm-io/alc0.compatible = ~
device-tree/arm-io/alc0.device_type = ~
device-tree/arm-io/alc2.compatible = ~
device-tree/arm-io/alc2.device_type = ~
device-tree/arm-io/aop/iop-aop-nub/aop-audio/audio-hpdbg.compatible = ~
device-tree/arm-io/aop/iop-aop-nub/aop-audio/audio-hpdbg.device_type = ~
device-tree/arm-io/aop/iop-aop-nub/aop-audio/audio-haptic.compatible = ~
device-tree/arm-io/aop/iop-aop-nub/aop-audio/audio-haptic.device_type = ~
And it works! We finally have a beta 2 kernel booted with the gIOServiceHideIOMedia flag set. And Xcode was able to mount DDI and fetch symbols as usual.
sh-5.0# ioreg | grep "busy [1-9]"
ioreg | grep "busy [1-9]"
sh-5.0# hfs: mounted DeveloperDiskImage on device disk2
static IOReturn AppleMobileFileIntegrityUserClient::loadTrustCache(OSObject *, void *, IOExternalMethodArguments *): PID 296 is requesting a trust cache load
AMFI: Successfully loaded a new image4 v1 trust cache with 78 entries.
sh-5.0# sw_vers
sw_vers
ProductName: iPhone OS
ProductVersion: 15.4
BuildVersion: 19E5219e
That brings us to CHARM - the Corellium Hypervisor for ARM, responsible for hardware virtualization.
Works like a CHARM
Since we know it is related to haptic, let's capture the entire kernel console log starting from the very beginning. Looking through it carefully we managed to spot the following failure:
AppleHapticsSupportLEAP::handleNewFW unable to create cal derived coeffs
AppleHapticsSupportLEAP::createSymbolFileAddressMap unable to find matching segment entry for load_address=0xf01310, length=1
[AOPAudio](error) condition "!(inFirmwareData->getLength() > maxLeapFirmwareBufferSizeBytes)" failed, AppleAOPAudioHapticLEAP::loadLeapFirmware() line 215, status: 0xe00002c2
[AOPAudio](error) expression "loadLeapFirmware(firmwareData)" failed, AppleAOPAudioHapticLEAP::externalClientRequestCallGated(), line 94, status: 0xe00002c2
The code around the error in the kernelcache looks like this:
v6 = (*(*v4 + 1448LL))(v4, v5, 0xC9LL, &v23, &v22);
if ( v6 )
{
v7 = v6;
v8 = (*(*a1 + 488LL))(a1, 0LL);
sub_FFFFFFF008F025C4(
"expression \"%s\" failed, %s::%s(), line %d, status: 0x%x\n",
"mController->getDeviceProperty(getIdentifier(static_cast<uint32_t>(LEAPIdentifierIndex::LEAPFirmwareAsset)), Apple"
"AOPAudioFirmware::FirmwareAssetProperties::kFirmwareBufferBytesMax, (void*)&maxLeapFirmwareBufferSizeBytes, &argSize)",
v8,
"loadLeapFirmware",
211LL,
v7);
return v7;
}
v11 = (*(*a2 + 160LL))(a2);
if ( v23 < v11 )
{
v9 = 0xE00002C2LL;
...
}
v23 is obtained from the Always-On Processor (AOP) as a response for 0xC9 command while v11 is a constant for the driver which equals 0xBF40
After all of that, it was simply a matter of handling yet another hardware request to make our virtual device model even more accurate.
Booting with the updated hypervisor we can confirm that no devices are busy and the flag is finally set in kernel.
iPhone:~ root# ioreg | grep "busy [1-9]"
iPhone:~ root# sw_vers
ProductName: iPhone OS
ProductVersion: 15.4
BuildVersion: 19E5219e
Conclusion: That's how to debug code
It was a long journey to narrow down the issue but hopefully we managed to demonstrate not only our approach to solve problems, but also various tools and technologies most of which are available for Corellium customers.
Keep reading
Introducing the Cortex-R82AE and Cortex-R82 System Models
Introducing MicroSnapshots - Leveraging copy-on-write memory
