ELMo Word Embeddings

This repository presents an analysis of different strategies for combining ELMo's contextual layer outputs and compares ELMo with traditional word embedding techniques on a classification task.

---

1. Role of λ Parameters in ELMo

In ELMo, each word representation is computed as a weighted sum of outputs from multiple layers. The weights (λ₁, λ₂, λ₃) determine the relative contribution of the token layer, the first LSTM layer, and the second LSTM layer.

---

1.1 Trainable λs

Configuration:

• λ values initialized uniformly: λ₁ = λ₂ = λ₃ = 0.3
• Optimized during training via backpropagation
• Softmax-like normalization applied to maintain a convex combination

Observed Behavior:

• Layer Specialization:

  • λ₂ (middle layer) became dominant: +1.1668
  • λ₃ (top layer) contributed negatively: –0.8541
  • λ₁ (token embeddings) remained small but positive: +0.0824

• Training Correlation:

  • Gradual divergence of λ values aligned with 98% reduction in loss
  • Final accuracy: 71.92%

Advantages:

• Learns task-specific layer importance
• Adjusts dynamically based on input and supervision

Limitations:

• Sensitive to initialization
• May exhibit instability in low-resource settings

---

1.2 Fixed λs

Configuration:

• Manually defined: λ₁ = 0.45, λ₂ = 0.15, λ₃ = 0.40
• Coefficients remain constant during training

Performance:

• Faster early convergence: +6.3% accuracy in first three epochs
• Final accuracy: 71.62%
• Layer contributions fixed throughout

Use Case Suitability:

• Reproducible baselines
• Scenarios with limited computational budget
• Applications prioritizing deterministic behavior

---

1.3 Learnable MLP Fusion

Architecture:

nn.Sequential(
    nn.Linear(3 * hidden_dim, hidden_dim),
    nn.ReLU(),
    nn.Linear(hidden_dim, hidden_dim)
)

Performance:

• Final accuracy: 71.64%
• Outperformed fixed λs
• Enhanced class-specific learning (e.g., +111 correct Class 3 predictions vs trainable λs)

Trade-offs:

• Higher computational cost
• Greater architectural tuning complexity

---

2. Classification Task Performance

2.1 Summary of Results

Method	Accuracy	F1 Score	Peak Epoch	Test Loss
Trainable λs	71.92%	0.7183	10	0.9171
Fixed λs	71.62%	0.7059	7	0.9554
Learnable MLP	71.64%	0.7164	9	0.9136

2.2 Confusion Pattern Insights

• Trainable λs: Effective separation of Classes 0 and 1
• MLP Fusion: Superior recall on Class 3
• Fixed λs: Balanced misclassification across classes

2.3 Training Stability

Method	Loss Fluctuation	Overfitting Onset
Trainable λs	Low	Epoch 8
Fixed λs	Medium	Epoch 5
Learnable MLP	Medium	Epoch 7

---

3. Comparison with Static Embedding Techniques

3.1 Benchmark Accuracy Comparison

Embedding Technique	Accuracy	Δ vs ELMo	Training Speed	Contextual Awareness
ELMo (Trainable)	71.92%	–	Moderate	Yes
CBOW	42.34%	–29.58%	Fast	No
SVD	42.59%	–29.33%	Fast	No

ELMo outperforms static methods by over 30%, attributed to its use of contextual representations.

3.2 Architectural Differences

• CBOW/SVD (Static):

  • One embedding per word
  • No position or context awareness
  • Struggles with polysemy and long-range semantics

• ELMo (Contextual):

  • Dynamic word representation based on context
  • Captures hierarchical linguistic features from stacked LSTM layers

3.3 Error Characteristics

Technique	Primary Error Pattern
CBOW	Uniform misclassification
SVD	Isolated misclassification spikes
ELMo	Class-boundary confusion

---

4. Conclusion

4.1 Layer Weighting Strategy Recommendations

Objective	Recommended Approach
Maximum classification accuracy	Trainable λs
Stable and interpretable baseline	Fixed λs (e.g., 0.45, 0.15, 0.40)
Class-sensitive representation	MLP-based layer fusion

4.2 Embedding Selection Guidelines

• Contextual embeddings such as ELMo should be prioritized in NLP pipelines

• Static embeddings may be used in:

  • Baseline evaluations
  • Low-compute environments
  • Situations where interpretability and reproducibility are critical

---

5. Model Hierarchy Summary

1. ELMo with Trainable λs — highest performance
2. ELMo with MLP Fusion — flexible, class-aware modeling
3. ELMo with Fixed λs — efficient and stable
4. Static Embeddings (CBOW, SVD) — fallback option