OmicsFUSION: A Pretrained Hyena-Based Framework for Encoding, Reconstruction, and Representation of the Omics and DNA Data

Abstract Recent advances in Natural Language Processing (NLP) have spurred the application of Large Language Models (LLMs) to bioinformatics, enabling innovative approaches to DNA sequence encoding. However, genomic function is not solely determined by the primary DNA sequence—it also depends on complex, multi-layered omics data. This auxiliary data is inherently sparse, structured across multiple tracks, and poses a challenge for traditional unimodal approaches. Simultaneously, many bioinformatics tasks demand a unified signal source that encapsulates this information, rather than requiring researchers to input each omics feature individually. To overcome these limitations, we introduce a novel framework for the integrated representation of DNA sequences and their associated omics data. Central to our approach is a large-scale, batch-structured omics dataset optimized for deep learning at scale. Our framework is built around four key novelties: (1) a new model architecture that jointly processes DNA and omics signals; (2) an extension of Masked Language Modeling (MLM) to omics tracks for effective data reconstruction; (3) single-nucleotide embeddings that fuse all input modalities; and (4) interval-level embeddings that summarize broader genomic regions. We release a collection of pretrained models capable of reconstructing, embedding, and generating unified representations of DNA enriched with functional omics context. Code https://github.com/aaai-2025-submission/anomymous_submission_aaai2025 Model and Datasets https://drive.google.com/drive/folders/10hcuq4rTCr8bKQFqkpz7R91MGtKvR0Va?usp=sharing