时间：12月14日（周四）上午 10：00 —— 11：00

地点：王选所 106 会议室

题目： The Locality and Symmetry of Positional Encodings

演讲者：Lihu Chen 

摘要： Positional Encodings (PEs) are used to inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models is not fully understood, especially given recent findings that various positional encodings are insensitive to word order. In this work, we conduct a systematic study of positional encodings in Bidirectional Masked Language Models (BERT-style), which complements existing work in three aspects: (1) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry; (2) We show that the two properties are closely correlated with the performances of downstream tasks; (3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly. We believe that these results are the basis for developing better PEs for transformer-based language models.

论文链接：https://arxiv.org/pdf/2310.12864.pdf

个人简介：Lihu Chen is a postdoctoral researcher working on NLP and data science at Inria (French Computer Science National Research).

His research is focused on Information Extraction and Large Language Models. Specifically, he is interested in the topics of entity linking, efficient language models, and detecting hallucinations in LLMs, as well as applications to the biomedical domain.

Lihu has worked at Max Planck Institute and Alibaba Group.

He completed his PhD at Institut Polytechnique de Paris (IP Paris), under the supervision of Prof. Fabian Suchanek and Dr. Gaël Varoquaux.