Structural Scaffolds for Making Sense of Document Collections (by Joe Barrow)
----
As readers, we often attempt to make sense of (one or more) documents using structure that goes beyond the content itself: a scientist using sections and subsections to "pre-read" a scientific paper or a web searcher trying to make sense of conflicting viewpoints about a topic. This structure helps a reader build mental maps of the information; without them, it is easy to "miss the forest for the trees." In this work, we aim to induce this structure in cases where it is not already explicit, which we refer to as "structural scaffolding." In particular, we focus on two types of scaffolds: topical scaffolds of documents, where we create labeled sections over an unstructured document to support pre-reading, and syntopical scaffolds of document collections, where we identify, group, and present viewpoints from many documents at once. We find that "content-only" approaches build worse scaffolds for both types than approaches that account for both content and context.

Tokenization in the era of pretrained language models (by Chenglei Si)
----
In this talk, we review the different tokenization strategies used in various pretrained language models (PLM). We focus on the granularity of tokenization (sub-character, character, sub-word, word) and compare their pros and cons in performance, efficiency and robustness. In particular, we highlight two of our own works along this line, one on fusing character and sub-word representations in English PLM, and the other on a novel sub-character tokenization scheme designed for Chinese PLM.

Joe will present over Zoom and Chenglei will present in person.
Zoom: https://umd.zoom.us/j/98806584197?pwd=SXBWOHE1cU9adFFKUmN2UVlwUEJXdz09
(passcode if needed: clip)