Creating Sequence Models and Transformers Course

Overview

Syllabus 19

1. Course Overview

   Get introduced to Creating Sequence Models and Transformers, meet your instructor, and discover the course objectives.

2. Why Sequence Models Matter

   Discover why the order of data matters, real-world applications of sequence models, key challenges, and foundational techniques for processing and modeling sequential information.

3. Breaking Text into Tokens

   Learn how text is broken into tokens, assigned numerical IDs, and prepared for AI models. Explore tokenization strategies and their impact on model performance and costs.

4. Creating a Tokenizer with PyTorch

   Learn how to build a tokenizer in PyTorch, transforming text into numerical data for NLP models, and compare simple word tokenization with advanced subword techniques used in modern AI.

5. Finding Meaning with Word Embeddings

   Learn how word embeddings transform words into vectors, letting machines grasp meaning, relationships, and analogies for smarter NLP applications like search, recommendations, and translation.

6. Pretrained Embeddings with PyTorch

   Learn to use pre-trained word embeddings in PyTorch, explore GloVe vectors, visualize relationships, solve analogies, and understand static versus contextual embeddings in NLP.

7. Sequence Modeling with RNNs, LSTMs, and GRUs

   Learn how RNNs, LSTMs, and GRUs model sequences, overcome memory limitations, and enable applications like text prediction, translation, and time series forecasting.

8. Predicting Sequences with RNN Variants in PyTorch

   Discover how RNN, LSTM, and GRU networks in PyTorch excel at predicting sequences, comparing their memory, accuracy, and efficiency in character-level text tasks.

9. Translating Ideas with Seq2Seq Models

   Explore how Seq2Seq models use encoder-decoder architectures to transform sequences, enabling tasks like translation, summarization, and chatbot responses.

10. Building Your First Seq2Seq Network with PyTorch

    Build a Seq2Seq model in PyTorch for tasks like translation and Q&A, learning about encoders, decoders, teacher forcing, and the context vector bottleneck.

11. Focusing Model Attention on What Matters

    Learn how attention mechanisms let AI models focus on relevant input parts, overcoming Seq2Seq bottlenecks and improving tasks like translation and summarization.

12. Enhancing Seq2Seq with Attention in PyTorch

    Learn to enhance Seq2Seq models in PyTorch using attention mechanisms, robust QA evaluation with EM and F1, and data-centric strategies to address complex language tasks.

13. Transformers and the Power of Self-Attention

    Explore Transformer architecture, self-attention, multi-head attention, and positional encoding to understand how modern AI models process language efficiently and contextually.

14. Exploring Transformers with HuggingFace

    Discover how Transformers process language by exploring tokenization, contextual embeddings, and attention visualization using HuggingFace and BERT for model interpretability.

15. Evaluating Transformer Models: Concepts and Industry Practices

    Learn to rigorously evaluate Transformer models using metrics like EM, F1, BLEU, and ROUGE, and qualitative error analysis for reliable, real-world language tasks.

16. Implementing Evaluation Metrics with Hugging Face

    Learn to implement and interpret evaluation metrics like EM, F1, Precision@k, and MRR for QA models using Hugging Face, and perform error analysis to guide model improvement.

17. Fine-Tuning Transformers for Q&A

    Learn to fine-tune pretrained transformers for extractive Q&A: prepare data, train with Hugging Face, evaluate using SQuAD, EM, and F1, and apply techniques in real-world scenarios.

18. Comparing RNNs and Transformers

    Compare RNNs (sequential processing) and Transformers (parallel self-attention) for sequence tasks in AI, focusing on strengths, limits, and real-world applications.

19. Project: Build a Transformer-Based Q&A System

    You will build a semantic retriever for an internal AI assistant. You’ll use transformer embeddings to encode queries and documents and return top-k results for LLM answers.

Skills you'll practise

Recurrent neural networksImplementing Word EmbeddingsTransformer neural networksWord embeddingsImplementing Attention MechanismsGenerative AI EvaluationImplementing Recurrent Neural NetworksHugging FaceImplementing Sequence-to-Sequence ModelsGenerative AI Evaluation ImplementationSequential text analysisSequence-to-Sequence ModelsNLP modelsImplementing TokenizationAttention mechanismsTokenizationImplementing Transformer Fine-Tuning

Advantages and disadvantages

Some points apply to every course of this kind; see how we rank.

Before you start

matplotlib. Jupyter notebooks. Matrix multiplication. Intermediate Python. NumPy. PyTorch. Feedforward neural networks. Pandas

Taught by

• Sohbet Dovranov