kleemans.ch

This semester, I'm following a Human-Computer-Interface lecture, and for our second project we had to evaluate an existing application. We decided to do a few user tests (as proposed by our brilliantly designed HCI book and website), and therefore capture all user input during the tests, including mouse movement positions, clicks, time, keyboard strokes.

As I didn't find an existing application (on Ubuntu) that served our approach, so I decided to put something simple together. As I did have some experience in "generating native system input events" with the Robot-class (as in this repo, which can generate mouse clicks based on a script), I decided to use Java. The problem is, to the contrary of generating input events, that Java on its own can't capture all input system-wide (only inside a AWT window or on a focussed element).

Luckily, there is a library jnativehook, which basically is a wrapper around some C++ code which adds a hook to the system for being able to capture all input. It works with the Java Native Interface (JNI) which allows C++ method calls (and other native interfaces).

I used some of the examples to put together a simple input capturer, which can be controlled with the shortkey CTRL + Q to save everything to logs and restart capturing. With the library, it's really easy. At first you'll have to register a hook at the "GlobalScreen":

GlobalScreen.registerNativeHook()

Then you just have to implement the methods given by the interfaces (NativeKeyListener):

public void nativeKeyPressed(NativeKeyEvent e) {
     System.out.println("Key Pressed: " + NativeKeyEvent.getKeyText(e.getKeyCode()));
}

At the end of a capture, 2 files are generated: summary.txt and positions.csv. The summary file contains all kestrokes, time duration and mouse distance. The positions.csv file simply contains a list of positions:

288,78
285,81
281,86
276,91
268,100
259,108
...

As we can export the position data directly as coordinate points, it's easy to draw them with a simple Python script. I exported the position data as CSV format for easy input reading and import it into python:

Read file:

with open("positions.csv", 'rb') as csvfile:
    reader = csv.reader(csvfile, delimiter=',', quotechar='|')
    coords = list(reader)

Plot coordinates:

im = Image.new('RGBA', screen, (255, 255, 255, 0)) # empty image
draw = ImageDraw.Draw(im) # draw object
last_pos = 0
for coord in coords:
    pos = (int(coord[0]), int(coord[1]))
    if (last_pos == 0):
        last_pos = pos
        continue
    draw.line([last_pos, pos], fill=line_color)
    last_pos = pos
im.save('path.png', 'PNG') # save

It exports a transparent image with the positions plotted as lines, so you can overlay them with a screenshot of the captured scene.

Some Examples

Angry birds:

Bejewelled:

Isoball:

Github Repo: https://github.com/akleemans/InputCapture