Proj. CAR Paper Reading: CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code

Abstract

本文：探索LLM在source code上pretrain时的subtokenization效果。
subtokenization: split long tokens into smaller subtokens, in order to ensure the relatively high frequency of all subtokens

效果：

1. 在下游任务表现没有明显下降的情况下，reduces average length by 17-40%
2. 如果特别挑选subtokenization，在可能导致length increase的情况下甚至可以增加0.5%-2%的表现

Base Model: PLBART

本栏目推荐文章
• 【略读论文|大模型相关】Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models
• 南阳师范大学 外国语学院 Foreign language school
• 大模型评测-微软亚洲研究院：A Survey on Evaluation of Large Language Models论文分享
• Language Models are Unsupervised Multitask Learners
• InstructGPT《InstructGPT: Training language models to follow instructions with human feedback》解读
• GPT-3《Language Models are Few-Shot Learners》解读
• GPT-2 《Language Models are Unsupervised Multitask Learners》解读
• GPT-1论文《Improving Language Understanding by Generative Pre-Training》解读
• 《OneLLM: One Framework to Align All Modalities with Language》论文学习
• 《Learning Transferable Visual Models From Natural Language Supervision》论文学习

Subtokenization Investigating Pretraining Language CodeBPEsubtokenization investigating pretraining language codebpe subtokenization pretraining investigating investigating property div-sum 11361 protein data-efficient蛋白质pretraining language vision-language accept-language