The impact of tokenizer selection in genomic language models

20253 citationsJournal Articlegold Open Access

Authors

LeAnn M. Lindsey · National Institutes of Health

Nicole L. Pershing · University of Utah

A K M Rubaiyat Reza Habib · University of Utah

Keith Dufault‐Thompson · National Institutes of Health

W. Zac Stephens · University of Utah

Anne J. Blaschke · University of Utah

Xiaofang Jiang

Abstract

Detailed results of all benchmarking experiments are available in https://github.com/leannmlindsey/DNAtokenization. Training datasets and pretrained models are available at https://huggingface.co/datasets/leannmlindsey. Datasets and processing scripts are available at doi: 10.5281/zenodo.16287401 and doi: 10.5281/zenodo.16287130.

Topics & Keywords

Topic Modeling

Publication Details

Published in: Bioinformatics

Volume 41, Issue 9

DOI: 10.1093/bioinformatics/btaf456

Field-Weighted Citation Impact: 7.54

Command Palette

The impact of tokenizer selection in genomic language models

Authors

Abstract

Topics & Keywords

Publication Details