One annoying CPU interpreter bug later and the thing is finally working.
The bug was in the ARMv5 signed halfword multiply and accumulate instruction that I added to my previous ARMv4 interpreter way back in december. It would accumulate the wrong value if the opcode accumulates the destination register into itself. At that time I apparently didn't think of this edge-case. I added a log message saying that it hit an experimental instruction though.
But when I started this emulator ~1 month ago I had the opportunity to run some tests for this instruction and concluded that it should be good and... removed that warning. If I had kept it around I probably would've noticed earlier that it uses this instruction and might have looked at it much earlier instead of wondering the whole day why the graphics are kinda broken >_<
I guess with something hard to debug like an emulator you can never be paranoid enough with warnings and assertions. This also reminds me that I should start building a proper debugger for this soon.