AI learns to hack the way children learn to ride a bicycle

Reinforcement learning sounds like something straight out of a military academy, but what exactly is it?
Pieter Libin: “It is a technique within artificial intelligence in which algorithms do not learn from a dataset, but through interaction with their environment. In fact, it is similar to the way people learn certain things. You can read in a book how to ride a bicycle, but ultimately you have to try it yourself. Naturally, you fall a few times at first, but gradually you learn how to keep your balance.
In scientific research, a major advantage is that such a learning process can be replicated in a simulator. In previous research projects, for instance, I used reinforcement learning to study strategies for epidemiological emergencies such as COVID-19. In the real world, you cannot test different strategies on the same pandemic, but in a simulation you can.”

Moving from an Atari game to the real world is still a big step.
Pieter: “Certainly, but after that things moved very quickly. In 2016 came AlphaGo, which learned to play the complex board game Go and defeated one of the world’s best players. Later, we saw agents learn to play even more complex video games such as StarCraft.
Go also demonstrated something else. An early version of AlphaGo was trained on historical data from human players. A later version became increasingly skilled purely by playing against itself. The agent played against a copy of itself, improved, and then played against that improved version. Remarkably, without human prior knowledge, the agent discovered new strategies that surprised human experts. That is scientifically very interesting. As humans, we have certain ways of thinking about a problem, but those ways can also limit our ability to see new solutions.”

“Our focus is defensive. We are developing algorithms that can help make our own infrastructure more secure"

How can reinforcement learning be used to improve the security of computer networks?
Pieter: “To do that, we simulate a computer network and allow reinforcement learning algorithms to develop strategies for penetrating the system. In this way, we want to investigate whether reinforcement learning can learn from scratch which known techniques are most effective for identifying vulnerabilities in common computer networks, and whether such an agent can eventually become as efficient as a human expert.”

You are collaborating with the Royal Military Academy in Brussels. How did that come about?
Pieter: “The Royal Military Academy collaborates with universities on doctoral research. One of their researchers, Raphaël Simon, had started his research under Professor Wim Mees at the RMA and subsequently contacted me to begin a PhD. The expertise required for his research aligned well with my own work on reinforcement learning and simulations.”

So you are teaching AI how to penetrate systems. Given the reports of hacking AI agents this summer, isn’t that dangerous?
Pieter: “Our focus is defensive. We are developing algorithms that can help make our own infrastructure more secure. An important difference from large language models is that our research takes place in a highly controlled and clearly defined environment.
With reinforcement learning, we determine exactly which actions an agent is allowed to perform. That gives us control over what the algorithm can do. With large language models, that is much more difficult. They are initially trained on vast amounts of text to predict the next word. As a result, they learn a wide range of things without a single, clearly defined objective. It is therefore extremely difficult to impose watertight safety boundaries on them after the fact.”

How does such an AI agent learn to test a computer network?
Raphaël: “We start with a simulated network. The agent must gather information about the systems within that network, identify vulnerabilities, and then decide what to do with that information. It becomes a cycle of information gathering and applying what it has learned. We begin with small, simple networks and gradually make them larger and more complex. To do that, we need fast simulators capable of generating a wide variety of experiences.”

Why are those fast simulations so important?
Pieter: “One of the main limitations of reinforcement learning is that an agent requires an enormous amount of experience. The agent performs an action, observes the outcome, and receives a reward indicating whether that action was good or bad. This process must be repeated many times before it learns how to act effectively.
With simple video games, you can simulate millions of interactions very quickly. An epidemiological model or a computer network model, however, is far more complex. That is why we are trying both to identify which algorithms work best and to make the simulator itself as fast as possible.”

But those simulators are not exactly the same as real computer networks, are they?
Raphaël: “No. In research environments, an agent often has access to all the information it needs. In the real world, that is not the case. You only have part of the information available. Moreover, we want what an agent learns to be useful not just for one specific network, but transferable to other networks as well.

“The closer you get to the real world, the more complex things become"

“That makes it very clear just how challenging it is to tune reinforcement learning systems and enable them to learn efficiently. We often say that they learn through interaction, much like humans do, but there is an important difference: an agent starts with no prior knowledge. A child learning to ride a bicycle has already seen other people cycling or has received advice from parents on how to stay upright. A reinforcement learning agent, in principle, tries everything. That is why such a vast number of interactions is required.”

Can you provide such an agent with prior knowledge?
Pieter: “That is an interesting research question. One possibility is to use language models to provide prior knowledge, while still keeping the learning process within the safe, controlled environment offered by reinforcement learning. Whether this would actually lead to better results, however, remains uncertain.
As I mentioned earlier, the AlphaGo agent that was not trained on historical data from human players ultimately outperformed the version that was. That finding could also be relevant in other domains. Of course, providing prior knowledge might accelerate learning, which would be an advantage.”

Could reinforcement learning discover entirely new cybersecurity strategies in this way? 
Raphaël: “In theory, yes. But it depends on the actions that are made available to the agent and therefore on how much room for creativity it is given. That immediately creates a tension: the more creativity you allow, the more important it becomes to build in clear boundaries and safeguards.”