DNA language models can learn reusable representations of genomic sequences that can later be adapted to downstream tasks. One of the central design choices in these models is how the sequence is represented and segmented before being processed by the model. Although nucleotide-level representations preserve base-level detail, they yield long input sequences and impose quadratic attention overhead as sequence length grows. Conversely, higher sequence compression lowers computational costs but risks sacrificing crucial local resolution.
In this work, we investigate Locally Consistent Parsing (LCP) as a segmentation strategy for DNA language models. LCP partitions sequences into variable-length, locally determined, and hierarchically organized segments. We develop an architecture that combines local nucleotide-level representations with LCP-based global representations, allowing the computationally expensive global attention layers to operate over a substantially shorter sequence while retaining access to base-level information. We pretrain the model using masked language modeling on human genomic sequence and evaluate it on the NT18 downstream benchmark. Our current experiments examine the feasibility of this architecture and motivate further study of the trade-off between sequence compression and fine-resolution information. We also discuss remaining limitations, ongoing ablations, and future directions.
Zoom Link: https://umd.zoom.us/j/