0. Abstract

• OriginalBERT : sentence semantic similarity에서 poor performance
  • 이유
    • static token embeddings biases and the ineffective BERT layers
    • NOT the high cosine similarity of the sentence embeddings
  • 모델
    • prompt based sentence embeddings method
      • reduce token embeddings biases
      • make the original BERT layers more effective
    • reformulating sentence embeddings task → fillin-the-blanks problem
    • 2 prompt representing methods and 3 prompt searching methods
  • Experiments
    • both non fine-tuned and fine- tuned settings
    • non fine-tuned method PromptBERT > unsupervised ConSERT on STS tasks
    • fine- tuned method PromptBERT > SOTA in SimCSE in both unsupervised and supervised settings

1. Introduction

• BERT의 성능이 안좋은게, anisotropy(이등방성) is not the main cause → the quality of sentence embeddings

  • simple experiment

    →by simply removing these biased tokens (e.g., high frequency subwords and punctuation) → using the average of the remaining token embeddings as sentence representation

  • 결론 : avoiding embedding bias can improve the performance of sentence representations.

  • 한계 : it is labor-intensive to manually remove embedding biases

→ 해결 : fill-in-the-blanks problems by different prompt

• Prompt based method can avoid embedding bias and utilize the original BERT layers.
• prompts : to generate positive pairs by different viewpoints from different templates
• 결론 : a prompt based contrastive learning method with template denoising to leverage

2. Related Works

(1) Contrastive Learning based methods → constructing positive sentence pairs.

(2) anisotropy : BERT-flow, BERT-whitenng → to reduce the anisotropy by post-processing the sentence embeddings from original BERT

3. Rethinking the Sentence Embeddings of Original BERT

• Semantic Similarity에서 BERT의 성능이 안좋은 이유는 aniostropy (X) → ineffective BERT layers and static token embedding biases

3-1) Observation1 : Original BERT layers fails to improve the performance

• BERT layer가 미치는 영향 분석 → two sentence embedding 비교

  (1) averaging static token embeddings (input of BERT)

  (2) averaging last layer (output of BERT)

  → measure the sentence level anisotropy

  

  : anisotropy of sentence embeddingd은 cosine similarity로 계산됨

  → 1에 가까울수록 the more anisotropy

  

• bert-base-uncased, roberta-base : harm the sentence embeddings performance

• 또한, performance degradation of BERT layers가 NOT sentence level anisotropy 때문

  → 이유 : last layer avg.가 static token 보다 더 isotropic