GameBoy emulator accuracy for Speedrunning

Goal: Determine the accuracy (in terms of runtime speed) of emulators and different hardware platforms to see if they are suitable for speedrunning.

Approach: Do many black-box tests (i.e. assume nothing about the implementation) using reproducible segments of real games and time how close each emulator/platform gets to the original hardware.

Results:

Notes:

This table mainly addresses the question whether a given emulator/platform can be considered "compatible" with original GameBoy hardware (time-wise), e.g. can be tracked on the same leaderboard.
If you are a runner and looking for an emulator recommendation, look for the "Recommended" tags in the list.
As we know for a long time, the original SGB runs about 2.4% too fast. Sometimes it is allowed on leaderboards nonetheless, but the run times must be converted, for example with nudua's converter.
One part of blargg's test fail when running on SGB2, some emulators in SGB2 mode also emulate this. This is indicated with a `OK*` for the corresponding test. As long as they show exactly the same behaviour as the original SGB2, they still pass blargg's tests.

Tests

Rationale

Tests must be consistent/deterministic. Repeated tests show the exact same results, also all original hardware behaves the same (don’t track hardware generation-related quirks like this).
Tests must be easy to execute. The segment to record must be fast to reach, and only consistent pressing of buttons throughout the test should happen (not even buffered inputs or similar) to avoid input errors.
Segment to record must be of reasonable length (> 1 minute) to and further to mitigate recording issues like frame fading.

Current test suite

Currently the following games are used for testing (with their respective column header in bold):

SML: Super Mario Land (World) (Rev 1) (sha1: 418203621b887caa090215d97e3f509b79affd3e)
KDL: Kirby's Dream Land (USA, Europe) (sha1: 90979baa1d0e24b41b5c304c5ddaf77450692d5a)
DT: Dick Tracy (USA) (sha1: 906361b2066c2b48500b9b709f7b4ed1018309c0)
BK: Balloon Kid (USA, Europe) (sha1: 0cb5adc9bdef5320f3f156efed0d47a618e2299f)
BB: Buster Bros. (USA) (sha1: 0d1692ff60ef1f6a97bbfd2bf8c1548f1f7439ed)
SML 2: Super Mario Land 2: 6 Golden Coins (USA, Europe) (Rev 2) (sha1: d11d94fa3c36b9f72e925070b66bb4f16d31001e)
TS: TaleSpin (USA) (sha1: 155eb5458c971a9a84e3be19711994ca2a4c24d8)
AS: Altered Space - A 3-D Alien Adventure (USA) (sha1: ca586c59b2473a8bec2f0f37504cb74e3a1e4d11)
DK: Donkey Kong (Japan, USA) (SGB Enhanced) (sha1: 6ed661bd1d6d8cdd48e1c10f8ca4e8dcba49128e)
WL2: Wario Land II (USA, Europe) (sha1: c65820b2e52d00e6ce60e0a432fab002fec4386f)

Each test is tracked in a separate sheet (accessible at the bottom of the spreadsheet). More details regarding frame number, execution date etc. can be found there.

Test execution instructions

Marker frames

Additional tests

Besides the black-box tests, a minimal set of white-box tests are executed, blargg's tests. The reason for this is to ensure a minimal level of confidence in the implementation of the emulator regarding CPU instruction/timing and memory timing.
blargg's tests can be found here: http://slack.net/~ant/old/gb-tests/. Sound tests are not required to pass.

Criteria for passing

To be considered "suitable for speedrunning", an emulator/platform has to fulfill the following criteria:

Pass blargg's tests (except sound tests)
Pass all tests within 0.2% deviation of original hardware, and show minimal deviation between multiple recordings (<= 0.1%).

Recordings

The following hardware and software was used for recording (unless mentioned differently):

Windows: OBS @ 60fps
macOS: OBS @ 60fps & Quicktime @ 60fps
Hardware: Huawei P20 lite @ 30.25fps (fixed framerate, see notes on test runs)
Raspi 4B: SimpleScreenRecorder @ 30 FPS

After recording, ffmpeg was used to add frame numbers to each frame. The marker frames, as defined in the document above, are then directly entered into the spreadsheet and the amount of frames and difference to the baseline (SGB2) is calculated.

Contact

Discord: adrianus#9213
Email: @gmail.com

All video footage is available for verification. Please contact me if you are interested.