Chapters
Chapters

Chapters

Hide chapters

Advanced Apple Debugging & Reverse Engineering

Fourth Edition · iOS 16, macOS 13.3 · Swift 5.8, Python 3 · Xcode 14

Before You Begin

Section 0: 6 chapters
Show chapters Hide chapters

Section I: Beginning LLDB Commands

Section 1: 10 chapters
Show chapters Hide chapters

Section II: Understanding Assembly

Section 2: 3 chapters
Show chapters Hide chapters

Section III: Low Level

Section 3: 6 chapters
Show chapters Hide chapters

Section IV: Custom LLDB Commands

Section 4: 8 chapters
Show chapters Hide chapters

Section V: DTrace

Section 5: 3 chapters
Show chapters Hide chapters

Appendices

Section 6: 3 chapters
Show chapters Hide chapters

14. System Calls & Ptrace
Written by Walter Tyree

Heads up... You're reading this book for free, with parts of this chapter shown beyond this point as scrambled text.

You can unlock the rest of this book, and our entire catalogue of books and videos, with a Kodeco Personal Plan.

Unlock now

As alluded to in the introduction to this book, debugging is not entirely about just fixing stuff. Debugging is the process of gaining a better understanding of what’s happening behind the scenes. In this chapter, you’ll explore the foundation of how debugging works, namely, 2 powerful APIs that enable lldb to attach and control a process. They are the mach exception setter APIs and the system call, ptrace.

In addition, you’ll learn some common security tricks developers use with ptrace to prevent a process from attaching to their programs. You’ll also learn some easy workarounds for these developer-imposed restrictions.

System Calls

Wait, wait, wait… ptrace is a system call. What’s a system call?

A system call is an entry point into code handled by the kernel. System calls are the foundation for userland code to do anything of interest, like opening a file, launching a process, consulting the value of a process identifier, etc.

From the userland side, a system call will marshal the appropriate arguments and send them over to the kernel. Userland code is not able to see the implementation details (i.e. the assembly) of what’s happening in the kernel. A userland function takes the arguments and executes a trap, think of it as a arm64 bl or x86_64 call instruction, to a function in the kernel. The kernel takes the arguments, determines if the arguments are well formed and if the process has permission to do the action. The kernel will then carry out that system call or deny accordingly.

For example, getpid, which gets the process identifier for the current process is actually a system call. The userland “source” to this is handwritten assembly found in xnu’s libsyscall/custom/__getpid.s. On the kernel side, the getpid call is picked up and eventually calls getpid(proc_t p, __unused struct getpid_args *uap, int32_t *retval) found in xnu’s bsd/kern/kern_prot.c.

Note: There are many unique system call wrappers that will call into the kernel, but there’s also a generic API to make system calls via the syscall(int, ...) function. One supplies an integer available from the <sys/syscall.h> header (or finds a private syscall number that’s in use) and passes in the expected arguments to that function. For example to mimic the __exit(int status) system call, you’d execute the syscall(SYS_exit, status); where status is the return value you’d pass into __exit.

Finding System Calls

To get a list of system calls, you can peruse the sources of xnu on opensource.apple.com. Alternatively, you can use DTrace to dynamically find them at runtime.

sudo dtrace -ln 'syscall:::entry' | wc -l

ptrace

With system calls explained, you’re now going to take a look at the ptrace system call in more depth. The easiest way to describe ptrace is that it enables setting certain debugging related flags for a process that are only accessible from the kernel side. This allows the debugger to catch the debugee if the debugee were to crash. For those interested at exploring the source, look at the ptrace kernel code to see what’s happening and search for references to P_LTRACED.

sudo dtrace -qn 'syscall::ptrace:entry { printf("%s(%d, %d, %d, %d) from %s\n", probefunc, arg0, arg1, arg2, arg3, execname); }'

while true {
  sleep(2)
  print("Hello ptrace!")
}


This request allows a process to gain control of an otherwise unrelated process and begin tracing it. It does not need any cooperation from the to-be-traced process, but the kernel does require the parent process to contain the right privileges. In this case, `pid` specifies the process ID of the to-be-traced process, and the other two arguments are ignored.

ptrace(13, 915, 5635, 0) from debugserver

Creating Attachment Issues

A process can actually specify it doesn’t want to be attached to by calling ptrace and supplying the PT_DENY_ATTACH argument. This is often used as an anti-debugging mechanism to prevent unwelcome reverse engineers from discovering a program’s internals.

ptrace(PT_DENY_ATTACH, 0, nil, 0)

Program ended with exit code: 45

Getting Around PT_DENY_ATTACH

Once a process executes ptrace with the PT_DENY_ATTACH argument, making an attachment greatly escalates in complexity. However, there’s a much easier way of getting around this problem.

sudo lldb -n "helloptrace" -w


(lldb) rb ptrace -s libsystem_kernel.dylib

(lldb) continue

(lldb) thread return 0

(lldb) continue

Other Anti-Debugging Techniques

Since we’re on the topic of anti-debugging, let’s put iTunes on the spot: for the longest time, iTunes actually used the ptrace’s PT_DENY_ATTACH. However, more recent versions of iTunes has opted for a different technique to prevent debugging: iTunes will now check if it’s being debugged using the powerful sysctl function, then kill itself if true. sysctl is another kernel function (like ptrace) that gets or sets kernel values. iTunes repeatedly calls sysctl while it’s running using a NSTimer to call out to the logic.

let mib = UnsafeMutablePointer<Int32>.allocate(capacity: 4)
mib[0] = CTL_KERN
mib[1] = KERN_PROC
mib[2] = KERN_PROC_PID
mib[3] = getpid()

var size: Int = MemoryLayout<kinfo_proc>.size
var info: kinfo_proc? = nil

sysctl(mib, 4, &info, &size, nil, 0)

if (info.unsafelyUnwrapped.kp_proc.p_flag & P_TRACED) > 0 {
  exit(1)
}

Key Points

  • ptrace is a system call that attaches to other processes.
  • Apps can deny ptrace attachments using the PT_DENY_ATTACH argument.

Where to Go From Here?

With the DTrace dumping script you used in this chapter, explore parts of your system and see when ptrace is called.

13. Assembly & the Stack
15. Shared Libraries
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2024 Kodeco Inc.

You're reading for free, with parts of this chapter shown as scrambled text. Unlock this book, and our entire catalogue of books and videos, with a Kodeco Personal Plan.

Unlock now