AI Luminaries
A curated collection of videos featuring top experts in artificial intelligence.
Overview
Centralized access to information and its subsequent processing is often computationally prohibitive over large networks due to communication overhead and the scale of the problem. Consequently, such systems rely on control and optimization algorithms that are fully distributed or even decentralized in nature. This course will provide a comprehensive overview of design and analysis of distributed optimization algorithms and their applications to machine learning. The aim is to revisit classical control and optimization algorithms for centralized optimization and discuss how these can be extended to distributed setting to accommodate the effects of communication constraints, network topology, computational resources, and robustness. Topics include graph theory, iterative methods for convex problems, synchronous and asynchronous setups, consensus algorithms, and distributed machine learning. We will also explore some recent literature in this area that exploits control theory for design of accelerated distributed optimization algorithms.
Syllabus 12
- Week 1.1: Introduction to Distributed Optimization
- Lecture 1: Introduction to optimization
- Lecture 2: Analyzing optimization algorithms in continuous time domain
- Lecture 3: Course Outline
- Lecture 4: Basics of optimization problems
- Lecture 5: Convex sets and Convex functions
- Week 2.1: Strong Convexity and Its Implications
- Lecture 6: Strictly and strongly convex functions
- Lecture 7: Implications of strong convexity
- Lecture 8: Primal and dual optimization problems
- Week 3
- Lecture 9: Slaters condition
- Lecture 10: Analysis of gradient descent algorithm
- Lecture 11: KKT conditions
- Week 4
- Lecture 12: Acceleration under strong convexity
- Lecture 13: Accelerate the convergence even further
- Lecture 14: Stability theory
- Lecture 15: Connections to optimization problems
- Week 5
- Lecture 16: Exponential stability
- Lecture 17: Bregman Divergance
- Lecture 18: Rescaled Gradient Flow
- Week 6
- Lecture 19: Advanced Results on PL inequality: Part 1
- Lecture 20: Advanced Results on PL inequality: Part 2
- Lecture 21: Constrained Optimization Problem
- Lecture 22: Augmented Lagrangian
- Week 7
- Lecture 23: Method of Multipliers
- Lecture 24: Dual Ascent and Dual Decomposition
- Lecture 25: ADMM Algorithm
- Week 8
- Lecture 26: Basics of Graph Theory
- Lecture 27: Basics of Graph Theory-2
- Lecture 28: Consensus and Average Consensus
- Lecture 29: Consensus and Average Consensus-2
- Week 9
- Lecture 30: Consensus Algorithms
- Lecture 31: Consensus Algorithms-Fixed time
- Lecture 32: Distributed Economic Dispatch Problem
- Lecture 33: Algorithm for Uncapacitated EDP
- Lecture 34: Capacitated EDP
- Week 10
- Lecture 35: Algorithms for Distributed Optimization
- Lecture 36: Algorithms for Distributed Optimization-2
- Lecture 37: Continuous-time Distributed Optimization Algorithms
- Lecture 38: Introduction to Neural Networks
- Lecture 39: Large Scale Machine Learning
- Week 11
- Lecture 40: Decentralized Stochastic Gradient Descent
- Lecture 41: Decentralized Stochastic Gradient Descent -2
- Lecture 42: Introduction to Federated Learning
- Lecture 43: FedAvg Algorithm
- Lecture 44: Convergence Analysis of FL
- Week 12
- Lecture 45: Sources of Computational Heterogenity in FL
- Lecture 46: Objective Inconsistency Problem
- Lecture 47: General Update Rule
Advantages and disadvantages
Advantages
- Taught by IIT and IISc professors, and it follows the Indian university syllabus closely.
- All videos and assignments are free on NPTEL and SWAYAM.
- The certificate is recognised by many Indian universities for credit transfer and by GATE aspirants.
- Great for GATE and semester exam preparation.
- Taught by Indian Institute of Technology Bombay, one of the strongest names in its field.
- Completely free.
- Self-paced: start any time.
- A clear syllabus (12 parts) you can see before you start.
Disadvantages
- The certificate needs a proctored exam at a centre, which has a fee.
- Recorded classroom lectures: thorough, but slower than made-for-online courses.
- New runs start on fixed dates (January and July).
- Learning is free, but the certificate costs money.
Some points apply to every course of this kind; see how we rank.
Free to learn
- Free: Every video and assignment is free on NPTEL and SWAYAM. Enrol when the next run opens.
- Certificate: Optional. It needs a proctored exam at a centre, which has a fee.
Before you start
A background in convex optimization and differential equations is preferred
Taught by
- Prof. Mayank BaranwalIIT Bombay
Indian Institute of Technology Bombay is in Tier 1: world-leading universities and India's top institutes of our institution ranking (96/100).
Similar courses
Generative AI Program
-Learn more and enroll in the program: https://online.stanford.edu/programs/generative-ai-technology-business-and-society-program Recent advancements in generative AI are reshaping industries, pushing technological boundaries, and revolutionizing creative processes. In this fast-changing landscape…
Artificial Intelligence
For more information about Stanford's Artificial Intelligence programs visit: https://stanford.io/ai Delve into the exciting world of Artificial Intelligence with insights and research from the world's top experts. This playlist offers a comprehensive journey, from foundational machine learning con…
Stanford CS229: Machine Learning led by Andrew Ng | Autumn 2018
Led by Andrew Ng, this course provides a broad introduction to machine learning and statistical pattern recognition. Topics include: supervised learning (generative/discriminative learning, parametric/non-parametric learning, neural networks, support vector machines); unsupervised learning (cluster…
MIT 6.034 Artificial Intelligence, Fall 2010
View the complete course: http://ocw.mit.edu/6-034F10 Instructor: Patrick Winston In these lectures, Prof. Patrick Winston introduces the 6.034 material from a conceptual, big-picture perspective. Topics include reasoning, search, constraints, learning, representations, architectures, and probabili…
Large Language Models (LLMs)
Explore the cutting-edge realm of Large Language Models (LLMs) with this expertly curated YouTube playlist. Featuring insights from industry leaders, educational content, and in-depth discussions, this selection highlights the latest advancements in LLM technology.