ProphetNet: Predicting Future N-gram for Sequence-to ...
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
Weizhen Qi1 , Yu Yan2, Yeyun Gong3, Dayiheng Liu4, Nan Duan3, Jiusheng Chen2, Ruofei Zhang2, Ming Zhou3 1University of Science and Technology of China, 2Microsoft, 3Microsoft Research Asia, 4Sichuan University 1weizhen@, 2{yyua, jiuchen, bzhang}@ 3{yegong, nanduan, mingzhou}@, 4losinuris@
Abstract
This paper presents a new sequence-tosequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-stepahead prediction in the traditional sequenceto-sequence model, the ProphetNet is optimized by n-step ahead prediction that predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large-scale dataset (160GB), respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.
1 Introduction
Large-scale pre-trained language models (Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019) and sequence-to-sequence models (Lewis et al., 2019; Song et al., 2019; Raffel et al., 2019) have achieved remarkable success in downstream tasks.
Autoregressive (AR) language modeling, which estimates the probability distribution of the text corpus, is widely used for sequence modeling and sequence-to-sequence (Seq2Seq) learning (Sutskever et al., 2014). Recently, it also becomes one of the successful self-supervised objectives for large-scale pre-training as used in GPT-
Work is done during internship at Microsoft Research Asia.
Equal contribution
2 (Radford et al., 2019). Specifically, given a
text sequence x = (x1, . . . , xT ), AR language
modeling factorizes the likelihood into a product
p(x) =
T t=1
p(xt|x ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- introducing recognia technical fidelity investments
- tesla annual report 2020 stocklight
- stock market prediction using machine learning classifiers
- stock market prediction through sentiment analysis of
- re configuring hybrid meetings moving
- an update to the economic outlook 2020 to 2030
- chapter 1 descriptive statistics for financial data
- nachiappan nagappan
- abstract researchgate
- the 2020 2023 business cycle mheda
Related searches
- level n books for boys
- gram per liter to ppm
- sequence to sequence model
- sequence to sequence learning
- sequence to sequence learning with neural networks
- convolutional sequence to sequence learning
- sequence to sequence modeling
- sequence to sequence lstm
- lstm sequence to sequence regression
- sequence to sequence model keras
- lstm sequence to sequence pytorch
- directions for sequence for kids