🟢 BERTweet: A Transformer Model for Sentiment Analysis on Twitter Data

AIVIA Project Engine Language Models & Generative AI
, , , , , , ,
1

BERTweet: A Transformer Model for Sentiment Analysis on Twitter Data

Objective

The primary objective of this project is to develop a sentiment analysis model using BERTweet, a pre-trained language model specifically designed for Twitter data. By leveraging the power of transformer architectures and fine-tuning BERTweet on a labeled dataset of tweets, you will build a robust model capable of classifying the sentiment of tweets as positive, negative, or neutral. This project provides hands-on experience with natural language processing (NLP) techniques, transformer models, and handling social media data, which is often noisy and unstructured.


Learning Outcomes

By completing this project, you will:

  • Understand the fundamentals of transformer architectures, focusing on BERT and its variants.
  • Gain proficiency in preprocessing Twitter data, including handling slang, emojis, hashtags, and misspellings.
  • Fine-tune pre-trained language models (specifically BERTweet) for downstream tasks.
  • Learn how to evaluate NLP models using appropriate metrics.
  • Implement data augmentation techniques to improve model performance.
  • Understand the challenges and techniques in sentiment analysis of social media data.

Prerequisites and Theoretical Foundations

1. Python Programming (Intermediate Level)

  • Data Structures: Lists, dictionaries, sets, tuples.
  • Control Flow: Loops, conditionals, functions.
  • Object-Oriented Programming: Classes, inheritance.
  • Libraries: Familiarity with Pandas, NumPy, and matplotlib.
Click to view Python code examples 
# Example of data manipulation with Pandas
import pandas as pd

# Reading CSV data
df = pd.read_csv('tweets.csv')

# Filtering data
positive_tweets = df[df['sentiment'] == 'positive']

# Basic statistics
print(df['sentiment'].value_counts())

2. Understanding of Natural Language Processing (NLP)

  • Text Preprocessing:
    • Tokenization, stopword removal, stemming, lemmatization.
  • Machine Learning in NLP:
    • Classification algorithms (e.g., Logistic Regression, SVM).
  • Evaluation Metrics:
    • Accuracy, Precision, Recall, F1-score, Confusion Matrix.
Click to view NLP concepts

  • Tokenization:

    • Breaking down text into individual words or subwords.

  • Stopword Removal:

    • Eliminating common words that do not contribute much to the meaning (e.g., ‘the’, ‘is’).

  • Stemming and Lemmatization:

    • Reducing words to their root form.

  • Evaluation Metrics:

    • Accuracy: (TP + TN) / (TP + TN + FP + FN)
    • Precision: TP / (TP + FP)
    • Recall: TP / (TP + FN)
    • F1-score: 2 * (Precision * Recall) / (Precision + Recall)

3. Familiarity with Transformer Models

  • Understanding BERT (Bidirectional Encoder Representations from Transformers):
    • Concept of bidirectionality.
    • Pre-training and fine-tuning stages.
  • BERT Variants:
    • DistilBERT, RoBERTa, BERTweet.
Click to view transformer concepts

  • Transformers:

    • Architecture that uses self-attention mechanisms to process sequences of data.

  • BERT:

    • Pre-trained on large corpus with masked language modeling and next sentence prediction tasks.

  • Fine-tuning:

    • Adapting the pre-trained model to specific downstream tasks like sentiment analysis.

  • BERTweet:

    • A version of BERT pre-trained on large-scale English tweets, capturing Twitter-specific language patterns.

4. Basic Understanding of PyTorch and Hugging Face Transformers

  • PyTorch:
    • Tensors, autograd, neural network modules.
  • Hugging Face Transformers Library:
    • Pre-trained models, tokenizers, model fine-tuning.
Click to view PyTorch code examples 
import torch
from transformers import BertModel, BertTokenizer

# Load pre-trained model and tokenizer
model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenize input
inputs = tokenizer("Hello, how are you?", return_tensors="pt")

# Get model outputs
outputs = model(**inputs)

Skills Gained

  • Data Preprocessing for Social Media: Handling noisy and unstructured text data from Twitter.
  • Implementing Transformer Models: Fine-tuning BERT-based models for classification tasks.
  • Model Evaluation: Using appropriate metrics and visualization tools to assess model performance.
  • Understanding Transfer Learning in NLP: Leveraging pre-trained models for downstream tasks.
  • Advanced NLP Techniques: Data augmentation, handling imbalanced datasets, and improving generalization.

Tools Required

  • Programming Language: Python 3.7+
  • Libraries:
    • Pandas: Data manipulation (pip install pandas)
    • NumPy: Numerical computations (pip install numpy)
    • Scikit-learn: Machine learning algorithms and evaluation metrics (pip install scikit-learn)
    • Matplotlib or Seaborn: Data visualization (pip install matplotlib seaborn)
    • PyTorch: Deep learning framework (pip install torch)
    • Hugging Face Transformers: Access to pre-trained transformer models (pip install transformers)
    • NLTK or SpaCy: NLP tools for preprocessing (pip install nltk spacy)
  • Datasets:
    • Sentiment140: A dataset with 1.6 million tweets labeled for sentiment.
    • Twitter API (optional): For collecting real-time tweets.
  • Integrated Development Environment (IDE):
    • Jupyter Notebook, VSCode, or PyCharm.

Steps and Tasks

1. Setup and Data Acquisition

Tasks:

  • Install Required Libraries:

    • Ensure all the necessary Python libraries are installed.

  • Obtain a Labeled Twitter Dataset:

    • Option 1: Use a publicly available dataset like Sentiment140.
    • Option 2: Collect tweets using the Twitter API and label them manually or use weak supervision.

Implementation:

# Install required libraries
!pip install pandas numpy scikit-learn matplotlib seaborn torch transformers nltk

# Download NLTK data
import nltk
nltk.download('punkt')
nltk.download('stopwords')
Data Acquisition Example

  • Using Sentiment140 Dataset:

    import pandas as pd

# Load the dataset
df = pd.read_csv('sentiment140.csv', encoding='latin-1', header=None)
df.columns = ['target', 'id', 'date', 'flag', 'user', 'text']

# Keep only the necessary columns
df = df[['text', 'target']]

# Map target values to sentiment labels
sentiment_mapping = {0: 'negative', 2: 'neutral', 4: 'positive'}
df['sentiment'] = df['target'].map(sentiment_mapping)
df = df[['text', 'sentiment']]

# Preview data
print(df.head())

  • Using Twitter API with Tweepy (Optional):

    !pip install tweepy

import tweepy

# Set up API credentials
consumer_key = 'your_consumer_key'
consumer_secret = 'your_consumer_secret'
access_token = 'your_access_token'
access_token_secret = 'your_access_token_secret'

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)

# Collect tweets
tweets = []
for tweet in tweepy.Cursor(api.search, q='happy', lang='en').items(100):
    tweets.append(tweet.text)

# Create DataFrame
df = pd.DataFrame(tweets, columns=['text'])

    Note: Replace API credentials with your actual keys and tokens.

2. Data Preprocessing

Tasks:

  • Clean the Data:

    • Remove URLs, mentions, hashtags, punctuations, and special characters.

  • Handle Emoticons and Emojis:

    • Convert them to text or remove them.

  • Tokenization:

    • Use the tokenizer corresponding to the pre-trained model (e.g., BERTweetTokenizer).

Implementation:

import re
import string

def clean_tweet(text):
    # Remove URLs
    text = re.sub(r'http\S+|www.\S+', '', text)
    # Remove mentions and hashtags
    text = re.sub(r'@\w+|#\w+', '', text)
    # Remove punctuations and numbers
    text = re.sub(r'[^A-Za-z\s]', '', text)
    # Convert to lowercase
    text = text.lower()
    return text

df['clean_text'] = df['text'].apply(clean_tweet)
Explanation

  • Regular Expressions (re module):

    • Used for pattern matching and text manipulation.

  • Data Cleaning Steps:

    • Removing URLs ensures that links do not interfere with the model.
    • Mentions (@user) and hashtags (#topic) can be removed or replaced based on the approach.
    • Converting to lowercase standardizes the text.

3. Exploratory Data Analysis (EDA)

Tasks:

  • Understand Data Distribution:

    • Check the balance of sentiment classes.

  • Visualize Common Words:

    • Use word clouds or bar plots for frequent words.

  • Analyze Tweet Lengths:

    • Understand the distribution of tweet lengths.

Implementation:

import seaborn as sns
import matplotlib.pyplot as plt

# Sentiment distribution
sns.countplot(x='sentiment', data=df)
plt.title('Sentiment Distribution')
plt.show()

# Tweet length distribution
df['tweet_length'] = df['clean_text'].apply(lambda x: len(x.split()))
sns.histplot(df['tweet_length'], bins=30)
plt.title('Tweet Length Distribution')
plt.show()
Visualization Examples

  • Word Cloud:

    from wordcloud import WordCloud

positive_text = ' '.join(df[df['sentiment'] == 'positive']['clean_text'])
wordcloud = WordCloud(max_font_size=50, max_words=100, background_color='white').generate(positive_text)

plt.figure()
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.title('Most Common Words in Positive Tweets')
plt.show()

4. Preparing Data for BERTweet

Tasks:

  • Load BERTweet Model and Tokenizer:

    • Use the pre-trained BERTweet model from Hugging Face.

  • Encode the Data:

    • Tokenize the tweets and convert them into input IDs and attention masks.

  • Create PyTorch Datasets and Dataloaders:

    • For efficient batching and training.

Implementation:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler

# Load BERTweet tokenizer
tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base", use_fast=False)

# Tokenize tweets
encoded_inputs = tokenizer(list(df['clean_text']), padding=True, truncation=True, max_length=128, return_tensors='pt')

# Convert labels to numeric
label_mapping = {'negative': 0, 'neutral': 1, 'positive': 2}
df['label'] = df['sentiment'].map(label_mapping)

# Create dataset
dataset = TensorDataset(encoded_inputs['input_ids'], encoded_inputs['attention_mask'], torch.tensor(df['label']))

# Split into training and validation sets
from sklearn.model_selection import train_test_split

train_dataset, val_dataset = train_test_split(dataset, test_size=0.1, random_state=42)

# Create dataloaders
train_loader = DataLoader(train_dataset, sampler=RandomSampler(train_dataset), batch_size=32)
val_loader = DataLoader(val_dataset, sampler=SequentialSampler(val_dataset), batch_size=32)
Explanation

  • AutoTokenizer and AutoModel:

    • AutoTokenizer and AutoModelForSequenceClassification automatically load the correct tokenizer and model based on the model name.

  • Tokenization Output:

    • input_ids: Token IDs for each token in the text.
    • attention_mask: Indicates which tokens are padding (0) and which are actual data (1).

  • TensorDataset:

    • Combines input IDs, attention masks, and labels into a dataset.

  • DataLoader:

    • Handles batching and shuffling of data during training.

5. Fine-tuning BERTweet for Sentiment Analysis

Tasks:

  • Load Pre-trained BERTweet Model:

    • Initialize the model for sequence classification.

  • Set Up Training Parameters:

    • Define optimizer, learning rate, and epochs.

  • Implement Training Loop:

    • Train the model using the training data.
    • Validate the model using the validation data.

Implementation:

from transformers import AutoModelForSequenceClassification, AdamW, get_linear_schedule_with_warmup

# Load pre-trained model
model = AutoModelForSequenceClassification.from_pretrained("vinai/bertweet-base", num_labels=3)

# Move model to GPU if available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

# Optimizer and learning rate scheduler
optimizer = AdamW(model.parameters(), lr=2e-5, eps=1e-8)

epochs = 3
total_steps = len(train_loader) * epochs

scheduler = get_linear_schedule_with_warmup(optimizer,
                                            num_warmup_steps=0,
                                            num_training_steps=total_steps)

# Training loop
for epoch in range(epochs):
    print(f'Epoch {epoch + 1}/{epochs}')
    model.train()
    total_train_loss = 0

    for batch in train_loader:
        b_input_ids, b_input_mask, b_labels = tuple(t.to(device) for t in batch)

        model.zero_grad()

        outputs = model(b_input_ids,
                        attention_mask=b_input_mask,
                        labels=b_labels)

        loss = outputs.loss
        logits = outputs.logits

        total_train_loss += loss.item()

        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        optimizer.step()
        scheduler.step()

    avg_train_loss = total_train_loss / len(train_loader)
    print(f'Average Training Loss: {avg_train_loss:.2f}')

    # Validation
    model.eval()
    total_eval_accuracy = 0
    total_eval_loss = 0

    for batch in val_loader:
        b_input_ids, b_input_mask, b_labels = tuple(t.to(device) for t in batch)

        with torch.no_grad():
            outputs = model(b_input_ids,
                            attention_mask=b_input_mask,
                            labels=b_labels)

        loss = outputs.loss
        logits = outputs.logits

        total_eval_loss += loss.item()

        # Calculate accuracy
        preds = torch.argmax(logits, dim=1).flatten()
        accuracy = (preds == b_labels).cpu().numpy().mean() * 100
        total_eval_accuracy += accuracy

    avg_val_accuracy = total_eval_accuracy / len(val_loader)
    avg_val_loss = total_eval_loss / len(val_loader)

    print(f'Validation Loss: {avg_val_loss:.2f}')
    print(f'Validation Accuracy: {avg_val_accuracy:.2f}%')
Explanation

  • AdamW Optimizer:

    • An improved version of Adam optimized for training transformers.

  • Learning Rate Scheduler:

    • get_linear_schedule_with_warmup gradually decreases the learning rate during training.

  • Gradient Clipping:

    • Prevents exploding gradients by capping the gradient norms.

  • Evaluation Mode:

    • Disables dropout and other training-specific layers.

6. Evaluating the Model

Tasks:

  • Calculate Evaluation Metrics:

    • Accuracy, Precision, Recall, F1-score.

  • Confusion Matrix:

    • Visualize model performance across classes.

  • Analyze Misclassifications:

    • Identify where the model is making errors.

Implementation:

from sklearn.metrics import classification_report, confusion_matrix
import numpy as np

# Collect all predictions and true labels
model.eval()
predictions, true_labels = [], []

for batch in val_loader:
    b_input_ids, b_input_mask, b_labels = tuple(t.to(device) for t in batch)

    with torch.no_grad():
        outputs = model(b_input_ids, attention_mask=b_input_mask)
    logits = outputs.logits
    preds = torch.argmax(logits, dim=1).flatten()

    predictions.extend(preds.cpu().numpy())
    true_labels.extend(b_labels.cpu().numpy())

# Classification report
print(classification_report(true_labels, predictions, target_names=['negative', 'neutral', 'positive']))

# Confusion matrix
cm = confusion_matrix(true_labels, predictions)
sns.heatmap(cm, annot=True, fmt='d', xticklabels=['negative', 'neutral', 'positive'], yticklabels=['negative', 'neutral', 'positive'])
plt.xlabel('Predicted')
plt.ylabel('True')
plt.title('Confusion Matrix')
plt.show()
Explanation

  • Classification Report:

    • Provides precision, recall, F1-score, and support for each class.

  • Confusion Matrix:

    • Shows the counts of true positives, false positives, true negatives, and false negatives.

  • Analyzing Errors:

    • Helps in understanding where the model struggles, informing further improvements.

7. Improving Model Performance

Tasks:

  • Data Augmentation:

    • Increase dataset size using techniques like synonym replacement or back-translation.

  • Handle Class Imbalance:

    • Use techniques like oversampling minority classes or weighted loss functions.

  • Hyperparameter Tuning:

    • Experiment with learning rates, batch sizes, and number of epochs.

Implementation:

# Example of class weights
from sklearn.utils.class_weight import compute_class_weight

class_weights = compute_class_weight('balanced', classes=np.unique(df['label']), y=df['label'])
class_weights = torch.tensor(class_weights, dtype=torch.float).to(device)

# Modify loss function to include class weights
outputs = model(b_input_ids,
                attention_mask=b_input_mask,
                labels=b_labels,
                return_dict=True)

loss_fn = torch.nn.CrossEntropyLoss(weight=class_weights)
loss = loss_fn(outputs.logits, b_labels)
Explanation

  • Class Weights:

    • Assign higher weights to minority classes to penalize misclassification more.

  • Data Augmentation Techniques:

    • Synonym Replacement: Replace words with their synonyms.
    • Back-Translation: Translate text to another language and back.

  • Hyperparameter Tuning:

    • Use libraries like Optuna or Ray Tune for systematic tuning.

8. Testing on Unseen Data

Tasks:

  • Collect New Tweets:

    • Use the Twitter API to fetch recent tweets.

  • Preprocess and Tokenize:

    • Apply the same preprocessing steps used during training.

  • Make Predictions:

    • Use the fine-tuned model to predict sentiments.

Implementation:

# Assume new_tweets is a list of raw tweets
new_tweets = ['I love this new phone!', 'This weather is terrible...', 'I am feeling okay today.']

# Preprocess
clean_new_tweets = [clean_tweet(tweet) for tweet in new_tweets]

# Tokenize
encoded_inputs = tokenizer(clean_new_tweets, padding=True, truncation=True, max_length=128, return_tensors='pt')
input_ids = encoded_inputs['input_ids'].to(device)
attention_mask = encoded_inputs['attention_mask'].to(device)

# Predict
model.eval()
with torch.no_grad():
    outputs = model(input_ids, attention_mask=attention_mask)
    logits = outputs.logits
    preds = torch.argmax(logits, dim=1).flatten()

# Map labels back to sentiments
label_mapping = {0: 'negative', 1: 'neutral', 2: 'positive'}
predicted_sentiments = [label_mapping[pred.item()] for pred in preds]

# Display results
for tweet, sentiment in zip(new_tweets, predicted_sentiments):
    print(f'Tweet: {tweet}')
    print(f'Predicted Sentiment: {sentiment}\n')

9. Deploying the Model (Optional)

Tasks:

  • Save the Model:

    • Save the fine-tuned model and tokenizer for future use.

  • Build a Simple Interface:

    • Create a web app using Streamlit or Flask.

  • Set Up a REST API:

    • Use FastAPI or Flask to create an endpoint for predictions.

Implementation:

# Save the model and tokenizer
model.save_pretrained('bertweet_sentiment_model')
tokenizer.save_pretrained('bertweet_sentiment_model')

# Example with Streamlit
!pip install streamlit

import streamlit as st

st.title('Twitter Sentiment Analysis with BERTweet')

user_input = st.text_input('Enter a tweet for sentiment analysis:')

if st.button('Analyze'):
    clean_input = clean_tweet(user_input)
    encoded_input = tokenizer(clean_input, return_tensors='pt', truncation=True, max_length=128)
    encoded_input = {key: val.to(device) for key, val in encoded_input.items()}
    with torch.no_grad():
        output = model(**encoded_input)
    pred = torch.argmax(output.logits, dim=1).item()
    sentiment = label_mapping[pred]
    st.write(f'Predicted Sentiment: {sentiment}')
Explanation

  • Saving Models:

    • Enables loading the model without retraining.

  • Web Application:

    • Streamlit provides an easy way to create interactive web apps for data science projects.

  • REST API:

    • Allows integration with other applications and services.

10. Next Steps and Enhancements

Suggestions:

  • Explore Other Transformer Models:

    • Compare BERTweet with models like RoBERTa, XLNet, or DistilBERT.

  • Multi-Lingual Support:

    • Use models like XLM-Roberta for sentiment analysis in different languages.

  • Domain Adaptation:

    • Fine-tune models on specific domains (e.g., finance, healthcare) for better performance.

  • Explainability:

    • Use tools like LIME or SHAP to interpret model predictions.

  • Handling Sarcasm and Irony:

    • Incorporate techniques to better detect sarcasm, which is challenging in sentiment analysis.

Additional Resources

Datasets

  • Sentiment140 Dataset: Link
  • SemEval-2017 Task 4: Link

Conclusion

In this project, you have:

  • Developed a sentiment analysis model using BERTweet, tailored for Twitter data.
  • Gained practical experience with transformer models and fine-tuning for NLP tasks.
  • Learned how to preprocess and handle noisy social media text data.
  • Implemented model evaluation and interpreted results to improve performance.
  • Explored challenges specific to sentiment analysis on Twitter, such as handling slang and emojis.

This foundational knowledge prepares you for more advanced topics in NLP and deep learning, such as:

  • Sequence-to-Sequence Models: For tasks like machine translation or text summarization.
  • Question Answering Systems: Building models that can understand and answer questions from text.
  • Multimodal Learning: Combining text data with images or audio for richer models.