dev, computing and games

Here is some C++:

compiled to x86.

Obvious disclaimer this is brittle, very platform and compiler dependent, will break if you look at it the wrong way.

How it works: this part

    if (fnChar[0] == 0xE9) // Strip off dummy call
    {
        realMain = fnChar + ((fnChar[2] << 8) | fnChar[1]) + 5;
    }

is because some compiler settings can mean your functions are called from jump thunks. For example, if you take the address of function "main", stored as fnChar, it gives you

Rather than the function body you'd expect, it's an entry in a sea of jumps. These are incremental linking jump thunks. While we could deal with this by disabling incremental linking, it's not too hard to robustify against. To look up what's the "real" address of main, the code looks at the offset of the jump-relative, adds it on plus 5 which is the length of the jmp operation.

After that is this part:

    for (int i = 0; i < INT_MAX; ++i) // Find offset to jump to
    {
        if (realMain[i + 0] == 0xCC && realMain[i + 1] == 0x68)
        {
            return reinterpret_cast<SimpleFn>(realMain + i + 1);
        }
    }

This seeks through main for some delimiter. The delimiter, in this case, is

Man, this is a garbage post. 0/10 effort

Wherever it finds the int 3 delimiter, that's the code we want to call. Could key off of only 0xCC but that's a very common term, it also happens to be used for setting up an argument to __CheckForDebuggerJustMyCode, so use the first byte of the subsequent push to dis-ambiguate.

And finally, the

exit(0);

is a cheesy trick to compensate for the fact that we didn't set up the stack frame properly. We'd run into this problem when jumping into the middle of almost any function (by messing with the function pointer). As far as the compiler's concerned, you're supposed to jump to the beginning of functions. By jumping into the middle, we skip a bunch of initialization, like the allocation of stack-allocated variables in function scope. And switching to global variables doesn't help for debug. When compiling for debug, the generated code allocates stack space automatically for you even if you have no local variables.

When main exits, it'll compare the stack pointer, and make sure it reflects base pointer + stack allocation to make sure nothing corrupted the value of the stack pointer somehow. This validation would catch a legitimate issue in this case. Bailing out with exit avoids the whole thing.

In all, this is a case where the compiler says you have code that's not reachable but it actually is.

The principle of this being possible is obvious to some people, but not all. I've had people say to me that "good compilers can always 100% statically know if code is reachable or not". It's not true, for C++ to x86 anyway. I tell them, "No, code is just code. At the end of the day, if code exists in a binary, you can execute it. Compilers can't exhaustively evaluate what generated code will do. They can't solve the halting problem" They don't understand what I mean and there's a gap in understanding so I hope a proof-by-example would explain, even if it's a super contrived and impractical one.

"But what if I enable optimizations?" If you enable optimizations, the code labeled "unreachable" will be actually missing from the binary. Yes that means the compiler is wrong. To work around I'd suggest adding some dummy control flow to tell the compiler the code is reachable.

This was tested using MSVC. GNU C has some additional semantics around gotos and labels (e.g., && operator) where you can encounter this more readily.

If you want to try the above code, here:

https://godbolt.org/z/saW4KnKqE

April 12th, 2022 at 8:09 am