Adrian Henkel

Summary
In this work I investigated the suitability of diffusion language models for protein design by leveraging the so called Inverse folding problem. This means that I trained a diffusion model (DiffuSeq) that is able to generate protein sequences with a specific 3D conformation. Some of the data analysis is also part of the ProstT5 paper. I also created a poster summing up the main findings of this thesis for the ISMB/ECCB 2023 conference. This can be found here.

Abstract of the thesis:
Generative diffusion models, like DALL-E and Imagen, have demonstrated impressive proficiency in controlled data generation, particularly in the realm of high-resolution imagery, guided by natural language text prompts. On a parallel track, AlphaFold2 has approached the threshold of resolving one of biology's most challenging tasks - the protein folding problem - by predicting the 3D structure of proteins with unprecedented accuracy. While the structural prediction of proteins constitutes a captivating domain, the inverse problem, namely the controlled generation of sequences that fold to a predefined 3-dimensional structure, constitutes an equally crucial scientific frontier in multiple domains. This thesis serves as a proof-of-concept, highlighting the deployment of diffusion language models (dLMs) towards the goal of controlled protein design, thereby addressing the inverse folding problem. Our workflow hinged on the application of DiffuSeq, a novel, classifier-free, sequence-to-sequence text generation method introduced by Gong et al. (2022). The structural information was previously encoded using the 3Di structural alphabet, introduced by Foldseek. This format allows the reduction of a protein's spatial conformation into an one-dimensional representation by projecting the three-dimensional characteristics of an amino acid onto one of the 20 discrete 3Di tokens. In our investigation, we first examined the dependency of the 3Di alphabet on three-state secondary structure, establishing a clear prevalence (>50%) of specific secondary structures for each 3Di alphabet letter. Thereafter, we contrasted the generative capabilities of the two Transformer architectures: Bidirectional Encoder Representations from Transformers (BERT) and Enhanced Transformer with Rotary Position Embedding (RoFormer). Next, we demonstrate the superior performance of dLMs over the semi-random baseline by consistently exceeding across all employed structure similarity metrics. However, it is important to note that, despite these promising results, we fell short of achieving the performance levels exemplified by the state-of-the-art method, ProteinMPNN. Nonetheless, given the discrete text nature of our conditional information, our findings underscore the potential of dLMs as powerful tools for the generation of sequences that effectively capture and leverage the biological context in protein design.

Summary:
deconvsurvR is a web tool for bulkRNA and single cell RNA sequencing data. The workflow contains data exploration, deconvolution, survival analysis and various machine learning algorithms. Here you can find the github repository.
This work is currently under revision and might be published in the future.

Abstract of the report:
The composition of immune cell types in tissues of different conditions is believed to influence the current status of diseases. With deconvsurvR we present a bulk and single cell analysis tool as shiny application. Besides the ability to determine immune cell type frequencies in bulk datasets using deconvolution methods, partly from the immundeconv package and in single cell dataset using clustering, the tool offers survival analysis and machine learning based on both bulk and single cell data. The results as well as the the input data can be exported as a customized result file for review and validation.

Summary:
In this work I have developed a simulation framework for federated learning. With that, I have evaluated three self designed communication efficient approaches on various image datasets and neural networks of different complexities. Here you can find the github repository also containing my thesis.

Abstract of the thesis:
Due to new data mining technologies and the resulting large amounts of data, automated analysis methods have to be used in order to gain valuable information about biological pathways, protein structure or tissue phenotype. Since conventional algorithms often lack the ability to generalize, machine learning, a methodology that aims to create iterative self learning computational models on centralized data, has recently provided significant success. Artificial neural networks are an advanced machine learning algorithm inspired by the biological brain, that consist of multiple interconnected layers. The training process aims to find the optimum model that achieves the best classification of the training data. Although this methodology has already been applied successfully in many fields, it is associated with privacy concerns if training is performed in a centralized manner. To tackle this challenge, federated learning was introduced, where multiple and possibly geographically independent clients collaboratively train a global model on their respective local dataset without sharing their data but only sending the model parameters to the server and vice-versa, enabling privacy-preserving model training. With increased model size, the network bandwidth that is mandatory, to exchange the model parameters between the server and the clients, becomes a bottleneck. In order to cope with this challenge, different communication-efficient approaches have been developed including gradient quantification, gradient sparsification and more local updates using the FederatedAveraging algorithm. In this thesis, we first present theoretical formulas to model the network bandwidth usage of each approach. Next, we develop a software framework to simulate a federated deep neural network training using the different communication-efficient approaches. Finally, we evaluate the approaches from the accuracy and network efficiency perspectives using multiple models and datasets under independent and identically distributed (IID) data across the clients.