Running complex sets of machine learning experiments is challenging and time-consuming due to the lack of a unified framework. This leaves researchers forced to spend time implementing necessary features such as parallelization, caching, and checkpointing themselves instead of focussing on their project. To simplify the process, we have developed MEMENTO, a Python package that is designed to aid researchers and data scientists in the efficient management and execution of computationally intensive experiments. MEMENTO has the capacity to streamline any experimental pipeline by providing a straightforward configuration matrix and the ability to concurrently run experiments across multiple threads.
Research has shown that many ML algorithms are vulnerable to adversarial attacks. In an adversarial attack, an attacker meticulously crafts adversarial examples by exploiting the ML model's weakness. By adding small perturbations to the benign instances, the adversary forces the model to produce erroneous predictions with high confidence. Investigating successful adversarial attacks contributes to understanding the model’s weaknesses, and adversarial defenses have been developed to detect these attacks.
New adversarial attacks and defenses are published frequently, making the task of benchmarking them increasingly tedious. To thoroughly evaluate an attack, a researcher would need to test the new attack against existing attacks on multiple models on multiple datasets under different conditions and while facing different existing defense strategies. This process can be drastically simplified using MEMENTO, particularly if most benchmark results are already stored and only the experiments regarding the new attack need to be run.
In this project, we will build upon a previous project developing a consistent framework integrating multiple attacks into MEMENTO. We will extend this framework, run all set up experiments, provide output reports and visualizations for the researcher to use, and showcase contributions in a research paper.
Undergraduate
Experimental setup (code), results, figures, text, and ultimately a publication
Experience in Python programming is essential. Prior understanding of machine learning concepts will be an advantage. No prior knowledge of adversarial learning and applicability domain is required.
No lab has been assigned to this project