8. Evolutionary Fuzzing

8.1. Overview

Evolutionary fuzzing is a search technique inspired by evolutionary biology (Darwin) which aims at converging towards the discovery of weaknesses. It uses genetic algorithms in order to produce successive generations of test cases populations. The test cases creation is not only based on classic methods but also on the feedback retrieved from the targets. In this context, the first population is the only one instantiated the traditional way. The ones that follow are spawned using five steps:

  1. Fitness score computation: each test case, member of the current population, is given a score which is function of some metrics (impact on the target, diversity, and so on), which are calculated by the entity in charge of the monitoring aspects.
  2. Probabilities of survival association: depending on the score computed in the previous step, a probability of survival is associated to each individual.
  3. Dice are rolled: using the probabilities of survival: weakest test cases are killed.
  4. Mutation: aims to modify a little bit each individuals (flip some bits for instance) to find local optimums.
  5. Cross-over: on the contrary, involves huge changes in order to find other optimums. It combines the test cases that are still alive in order to generate even better solutions. This process is also used to compensate the kills done in step 3.
_images/evolutionary_process.png

Evolutionary process

The implementation within Fuddly is divided into three main components:

8.2. User interface

An evolutionary process can be configured by extending the framework.evolutionary_helpers.Population and framework.evolutionary_helpers.Individual abstract classes. These elements describe the contract that needs to be satisfied in order for the evolutionary process to get running. Basically, the methods _initialize() and reset() can be used to initialize the first population, evolve() to get the population to the next generation and is_final() to specify a stop criteria.

As these are very generic, they bring a lot of flexibility but require some work. To address this issue, Fuddly also proposes a default implementation that describes the classic approach introduced in the previous section. Each step is expressed using one of the framework.evolutionary_helpers.DefaultPopulation methods. The evolution stops when the population extincts or if a maximum number of generation exceeds.

  • _compute_scores(): computes the individuals fitness scores, which is, in the default implementation, a random score between 0 and 100. This implementation have to be overridden to match the context. Indeed, this method is used to characterize the adaptation of each test case to the target, meaning the negative impact it had on the target. Besides, it also deals with the diversity of the population in order to avoid its premature extinction.
  • _compute_probability_of_survival(): simply normalize fitness scores between 0 and 1.
  • _kill(): rolls the dices !
  • _mutate(): operates three bit flips on each individual using the stateless disruptor C.
  • _crossover(): compensates the kills through the use of the stateful disruptor tCOMB. Of course, any other disruptor could have been chosen (those introduced by the evolutionary fuzzing are described in the next section).

Finally, to make an evolutionary process available to the framework, it has to be registered at project level (meaning inside a *_proj.py file), through framework.Project.register_evolutionary_process(). This method expects processes in the form of 3-tuples containing:

Here under is provided an example to register an evolutionary process (defined in tuto_proj.py):

from framework.evolutionary_helpers import DefaultPopulation

project.register_evolutionary_processes(
    ('evol',
     DefaultPopulation,
     {'init_process': [('SEPARATOR', UI(random=True)), 'tTYPE'],
      'size': 10,
      'max_generation_nb': 10})
)

Once loaded from Fuddly, Scenario are created from registered evolutionary processes, which are callable (like any other scenarios) through their associated Generator. In our example, only one process is registered and will lead to the creation of the generator SC_EVOL. After each call to it, the evolutionary process will progress and a new test case will be produced.

Note that the framework.evolutionary_helpers.DefaultPopulation is used with this scenario. It expects three parameters:

  • The first one describe the process to follow to generate the data in the initial population (refer to the API documentation for more information). In the example, we use the generator SEPARATOR to produce data compliant to the model in a randome way, then we apply the disruptor tTYPE.
  • The second specify the size of the population.
  • The third is a criteria to stop the evolutionary process. It provides the maximum number of generation to reach

8.3. Specific disruptors

The evolutionary fuzzing introduces two stateful disruptors that can be used within the crossover operation.

8.3.1. tCROSS - Randomly swap some leaf nodes

Description:
Produce two children by making two graphs swap a given percentages of their leaf nodes.
_images/sd_crossover.png

tCROSS example

Reference:
framework.generic_data_makers.sd_crossover
Parameters:
generic args:
  |_ clone_node
  |      | desc: if True the dmaker will always return a copy of the node. (for
  |      |       stateless diruptors dealing with big data it can be usefull
  |      |       to it to False)
  |      | default: True [type: bool]
  |_ init
  |      | desc: make the model walker ignore all the steps until the provided
  |      |       one
  |      | default: 1 [type: int]
  |_ max_steps
  |      | desc: maximum number of steps (-1 means until the end)
  |      | default: -1 [type: int]
  |_ runs_per_node
  |      | desc: maximum number of test cases for a single node (-1 means until
  |      |       the end)
  |      | default: -1 [type: int]
specific args:
  |_ node
  |      | desc: node to crossover with
  |      | default: None [type: Node]
  |_ percentage_to_share
  |      | desc: percentage of the base node to share
  |      | default: 0.50 [type: float]

8.3.2. tCOMB - Randomly swap some root nodes’ children

Description:
Produce two nodes by swapping some of the children of two given graphs roots.
_images/sd_combine.png

tCOMB example

Reference:
framework.generic_data_makers.sd_combine
Parameters:
generic args:
  |_ clone_node
  |      | desc: if True the dmaker will always return a copy of the node. (for
  |      |       stateless diruptors dealing with big data it can be usefull
  |      |       to it to False)
  |      | default: True [type: bool]
  |_ init
  |      | desc: make the model walker ignore all the steps until the provided
  |      |       one
  |      | default: 1 [type: int]
  |_ max_steps
  |      | desc: maximum number of steps (-1 means until the end)
  |      | default: -1 [type: int]
  |_ runs_per_node
  |      | desc: maximum number of test cases for a single node (-1 means until
  |      |       the end)
  |      | default: -1 [type: int]
specific args:
  |_ node
  |      | desc: node to combine with
  |      | default: None [type: Node]