A recurrent neural network and the unfolding in time of the computation involved in its

Supervisor: Germain PHAM

Project topics : Machine learning, Optimization

Backpropagation Through Time (BPTT) is a powerful algorithm for training recurrent neural networks (RNNs), which are a type of neural network that can learn to process sequential data [Brownlee20]. RNNs are used in a wide range of applications, such as natural language processing, machine translation, and speech recognition.

Project description

While there are many online resources that claim to derive the BPTT equations, most of them are not rigorous or complete [Sidharth23][Sidharth23][G4G20][Krishna22][dsgiitr19][Kumar22].

The goal of this project is to fill this gap by providing a comprehensive and rigorous mathematical derivation of the BPTT algorithm. A critical part of this project will be to explain the derivation in a clear and precise manner and to produce a document that can be used as a teaching resource. The document will be written using AsciiDoctor, a fast, open source, text processor [AsciiDoctor]. Once the teaching resource has been produced, the algorithm will be implemented in Python or Matlab and tested on a simple task, such as learning to predict time series values.

This project is a great choice for students because it provides an opportunity to learn about and implement a fundamental algorithm in machine learning. The project also requires students to have a strong understanding of mathematics and programming.

In addition, the project is challenging because it requires students to derive the BPTT equations in a rigorous and complete manner. This is a task that is not typically done in machine learning courses and, explaining and providing the derivation of equations for algorithms is sometimes more demanding than simply implementing them. Furthermore, it will require to typeset the equations in LaTeX and to produce figures that illustrate the algorithm which is also not typically taught at school.

Required skills

This project requires a good knowledge of basic machine learning concepts, such as neural networks, gradient descent, and backpropagation.

  • Mandatory

    • Latex programming experience

    • practical elements of git

    • Linux OS basics (usage of terminal command lines)

    • enjoy writing equations !

  • Optional

    • basic elements on docker

Workplan (5 weeks)

gantt
title ICS Project: Teaching BPTT
dateFormat DD/MM/YYYY
excludes weekends
axisFormat %e %b %y
tickInterval 1week
weekday monday
todayMarker off

section Week1 - ANN Basics
Bibliography analysis - RNN and their optimization: 26/02/2024, 10d
Setup AsciiDoctor on Linux: 26/02/2024, 1d
ANN Basics - MLP + BackProp: 27/02/2024, 4d
Deliverable slides - BackProp presentation: milestone, 01/03/2024, 1d
section Week2 - RNN
RNN and BPTT: 04/03/2024, 1d
GRU and LSTM - Truncated BPTT equations: 05/03/2024, 4d
Deliverable - handwritten equations: milestone, 08/03/2024, 1d
section Week3 - Produce teaching material
Write equations derivation using AsciiDoctor and Latex: 11/03/2024, 5d
Deliverable report - typeset equations: 15/03/2024, 5d
section Week5 - Open source contribution
Code refactoring and documentation: 25/03/2024, 2d
Gitlab public project: milestone, 27/03/2024, 1d
Slides preparation: 28/03/2024, 1d
Project defense: milestone, 29/03/2024, 1d

Location

School

Télécom Paris trains its students to innovate in today’s digital world. Its training and research cover all fields of information and communication sciences and technologies with a strong societal foundation in order to address the major challenges of the 21st century. Its offers engineering, PhD and professional degree programs, with international students accounting for 55% of its student body. Its research offers original, multidisciplinary world-class expertise in nine strategic areas: Data Science and Artificial Intelligence — Visual and Audio Computing, Interaction — Digital Trust — Innovation Regulations — Transformation of Innovative Firms — Cyber-Physical Systems — Communication Systems and Networks — Mathematics and Applications — Uses, Participation, Democratization of Innovation.

As a founding member of Institut Polytechnique de Paris and an IMT (Institut Mines-Télécom) school, Télécom Paris is a living laboratory that fosters practical solutions and applications while measuring their impact on society.

location_on Address: 19 place Marguerite Perey, 91120 Palaiseau, France

Find us on Google Maps

Hosting laboratory

Laboratoire Traitement et Communication de l’Information (LTCI)

Research team

The Circuits et Systèmes de Communication (C2S) team is internationally recognized for its ability to integrate digital intelligence into AMS and RF SoCs such as analog-to-digital converters (ADCs) or RF receivers for cognitive radio. By combining its expertise in the physical realization of the CMOS chip with its experience in signal processing and its knowledge of the other network layers for which LTCI’s skills are recognized, the group designs high-performance AMS and RF SoCs. The aim is to develop elements or "building blocks", enabling the system of connected objects to be interfaced on one side with the physical world via sensors, and on the other side with the system core via communications, in particular RF.

References