Out-of-bounds: an outside view of computer memory vulnerabilities

Linus Hafkemeyer certainly isn’t the first who, as a teenager, tried to circumvent the restrictions on his internet use set by his parents and their router. But where others may grow bored with such tinkering, he became ever more fascinated by cybersecurity, ultimately leading up to a master’s thesis on one of the most hazardous types of memory corruption vulnerabilities. 

‘They are called out-of-bounds write vulnerabilities, and they can be used to trick a program into accessing memory at certain locations where that should not be possible,’ Hafkemeyer says. ‘If such a write ends up in unused memory, which will never be accessed again, it will not have any consequences. In many other cases, it will just make the program crash. But in certain cases, it could allow a hacker to, for example, modify critical data or even hijack an entire program.’

Abundant and reliable

What makes out-of-bounds writes so very dangerous is that they are both abundant and reliable. Once someone with bad intentions has developed a method to exploit such a vulnerability, it is very likely to succeed any time the program is being executed. Hafkemeyer: ‘I really got to know these vulnerabilities early on during my master’s, when participating in so-called Capture the Flag challenges – an information security competition in which you, for example, must exploit a vulnerability on a server to obtain a secret and score points.’ 

Dead set on his research topic, and with it hardly being taught and researched at TU Delft, he organised external supervision by Professor Andrea Continella from the University of Twente. Hafkemeyer: ‘He suggested some research directions, which I then narrowed down until I found a sub-problem that was not yet solved while still being feasible as a master’s thesis project.’ 

From the outside looking in

‘The problem with existing out-of-bounds write detection solutions is that they are invasive; they need to add some code to a program to facilitate detection,’ Hafkemeyer says. ‘But then, when observing program behaviour with the added code, it will differ from how the original, unaltered program would perform, making any output worthless for triaging.’ In an emergency room setting, it would be like triaging patients by having them randomly point at a list of physical complaints.

‘I designed an approach in which we run the unaltered program in a virtual machine, feeding it the input previously found through fuzzing that we suspect of causing an out-of-bound write,’ Hafkemeyer says. ‘This way, we can observe its behaviour from an outsider perspective and see what is going on without having to modify the program.’ His tool returns all the source code-level objects affected by such unintended writes, providing human experts precious information for triaging and even root-cause analysis.

Insanely complex

‘Looking back, I am amazed by the twists and turns this project took from the original idea to the finished thesis,’ Hafkemeyer says. ‘The more I got into the topic, the more I realized how insanely complex it is.’ Central to his difficulties was compilation – the process of translating the legible source code into an executable program. ‘This process is generally accompanied by a loss of most high-level semantic program information that is necessary for detecting out-of-bounds writes. I had to delve deeply into the inner workings of the compiler and develop ways to pull out the necessary information before it will be lost.’  

Computer programmers often talk about “being in the zone” – a hyper-focused and very productive state-of-mind. Next to this zone, Hafkemeyer at times also felt like being in his own tunnel, unsure if what he did made sense. ‘What helped me at those moments was to have a sanity check, presenting what I did to my friends and my supervisor,’ he says. ‘During my thesis work, I also got into historical audiobooks. They allowed me to dive into another reality, even if just for half an hour. And then return to the task at hand with renewed energy and insights.’