Deep Learning Lectures

class: center, middle

# Introduction to Deep Learning

Edouard Yvinec

.affiliations[
  ![Sorbonne](images/logo_sorbonne.png)
  ![Isir](images/logo_isir.png)
  ![Datakalab](images/logo_datakalab.png)
  ![Epita](images/Epita.png)
]

---

# Website

.center[
### https://edouardyvinec.netlify.app/posts/slides/
  ]
--
.center[
### https://deepcourse-epita.netlify.app/
]

---
# Goal of the class

## Overview

- When and where to use DL
- "How" it works
- Frontiers of DL

## Arcanes of DL

- Implement using `Numpy`, and `Tensorflow` (`Keras`)
- Engineering knowledge for building and training DL

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# What is Deep Learning

### Good old Neural Networks, with more layers/modules

### Non-linear, hierarchical, abstract representations of data

### Flexible models with any input/output type and size

### Differentiable Functional Programming

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Why Deep Learning Now?

- Better algorithms & understanding

- .grey[Computing power (GPUs, TPUs, ...)]

- .grey[Data with labels]

- .grey[Open source tools and models]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Why Deep Learning Now?

- Better algorithms & understanding

- Computing power (GPUs, TPUs, ...)

- .grey[Data with labels]

- .grey[Open source tools and models]

.center[
<img src="images/gpu_tpu.png" style="width: 450px;" /><br/><br/>
<small>_GPU and TPU_</small>
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Why Deep Learning Now?

- Better algorithms & understanding

- Computing power (GPUs, TPUs, ...)

- Data with labels

- .grey[Open source tools and models]

.center[
<img src="images/ng_data_perf.svg" style="width: 400px;" /><br/><br/>
<small>_Adapted from Andrew Ng_</small>
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Why Deep Learning Now?

- Better algorithms & understanding

- Computing power (GPUs, TPUs, ...)

- Data with labels

- Open source tools and models

.center[
<img src="images/frameworks.png" style="width: 500px;" /><br/><br/>
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: Speech-to-Text

.center[
<img src="images/speech.png" style="width: 780px;" />
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: Vision

.center[
<img src="images/vision.png" style="width: 720px;" />
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: Vision

.center[
<img src="images/vision2.png" style="width: 720px;" />
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: NLP

.center[
<img src="images/nlp.png" style="width: 600px;" />
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: NLP

.center[
<img src="images/nlp2.png" style="width: 720px;" />
]

Most of chatbots claiming "AI" do not use Deep Learning (yet?)

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: Vision + NLP

.center[
<img src="images/nlp_vision.png" style="width: 760px;" />
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: Image translation

.center[
<img src="images/vision_translation.png" style="width: 700px;" />
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: Generative models

.center[
<img src="images/nvidia_celeb.jpg" style="width: 350px;" />
<br/>Sampled celebrities [Nvidia 2017]
]

--
<br/>

.center[
<img src="images/stackgan.jpg" style="width: 600px;" />
<br/>StackGAN v2 [Zhang 2017]
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL Today: Generative models
.center[
<img src="images/WaveNet.gif" style="width: 400px;" />
<br/>Sound generation with WaveNet [DeepMind 2017]
]

Guess which one is generated?

.center[
<audio controls><source src="images/columbia_gen.wav"></audio> <br/>

<small>_Tacotron 2 Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions, 2017_</small>
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Language / Image models

Open-AI GPT-3, or DALL-E: https://openai.com/blog/dall-e/

.center[
<img src="images/dalle.png" style="width: 600px;" />
]
<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL in Science: Genomics

.center[

<img src="images/deepgenomics.png" style="width: 580px;" />
]

.center[
<img src="images/protein_fold.gif" style="width: 320px;" /><br/>
<small>[AlphaFold by DeepMind](https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology)</small>
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---

# DL in Science: Chemistry, Physics

.center[
<img src="images/deep_other.png" style="width: 680px;" />
]

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL in Science: Chemistry, Physics

.center[
<img src="images/Accelerating_Eulerian_Fluid_Simulation_with_Convolutional_Networks.gif" style="width: 350px;" />
]

- Finite element simulator accelerated (~100 fold) by a 3D convolutional network

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# DL for AI in games

.center[
<img src="images/games.png" style="width: 600px;" />
]

<small> AlphaGo/Zero: Monte Carlo Tree Search, Deep Reinforcement Learning, self-play </small>

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Outline of the class

### Neural Networks (2)

### Computer Vision (3)

### Few Shot and Unsupervised Learning  (2)

### Generative models (2)

### New Architectures (2)

---

# How this unit works

#### Lectures 1h-1h30

- Can include a Quiz on Moodle (from time to time)
- Small part of the final grade

#### Coding sessions 2h-2h30

- BYO laptop, you can work in pairs
- Homework every session (finish notebooks, read solutions)

#### Final Project

- Project of your choice in teams of 2-4 people.
- Contact me via e-mail (ey@datakalab.com) to select a topic as early as possible.

---
# Recommended reading

- [deeplearningbook.org](http://www.deeplearningbook.org/): Math and main concepts

- [Francois Chollet's book](https://www.manning.com/books/deep-learning-with-python): Keras programming

- [Aurélien Géron's book](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/):
  Generic Machine Learning with Scikit-learn and Deep Learning with TF/Keras

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
class: center,middle
# Frameworks and Computation Graphs

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Libraries & Frameworks

.center[
<img src="images/frameworks.png" style="width: 600px;" /><br/><br/>
]

This lecture is using **Pytorch**: high level frontend supported by facebook.

If you are familiar with tensorflow/keras, feel free to use these frameworks instead.

---
# Computation Graph

.center[
<img src="images/computation_graph_simple_f.png" style="width: 600px;" /><br/><br/>
]

Neural network = parametrized, non-linear function

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Computation Graph

.center[
<img src="images/computation_graph_simple.png" style="width: 600px;" /><br/><br/>
]

Computation graph: Directed graph of functions, depending on parameters (neuron weights)
<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Computation Graph

.center[
<img src="images/computation_graph_simple_expl.png" style="width: 600px;" /><br/><br/>
]

Combination of linear (parametrized) and non-linear functions
<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Computation Graph

.center[
<img src="images/computation_graph_complicated.png" style="width: 600px;" /><br/><br/>
]

Not only sequential application of functions
<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Computation Graph

.center[
<img src="images/computation_graph_backprop.png" style="width: 600px;" /><br/><br/>
]

Automatic computation of gradients: all modules are **differentiable**!

Theano (now Aesara), **Tensorflow 1**, etc. build a static computation graph via static declarations.

**Tensorflow 2**, **PyTorch**, **JAX**, etc. rely on dynamic differentiable modules: "define-by-run".

Vector computation on **CPU** and accelerators (**GPU** and **TPU**).

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>
---
# Computation Graph

.center[
<img src="images/computation_graph_backprop.png" style="width: 600px;" /><br/><br/>
]

Simple keras implementation

```py
model = Sequential()
model.add(Dense(H, input_dim=N))  # defines W0
model.add(Activation("tanh"))
model.add(Dense(K))               # defines W1
model.add(Activation("softmax"))
```

<div style="position: absolute; bottom: 5px; ">
<credits>credit to Olivier Grisel and Charles Ollion</credits>
</div>

---
# Internship Offers (2022-2023):
Datakalab proposes 3 internships for software engineers and 1 internship for hardware engineer:

<small>
- python engineer: product implementation of the research work from our lab (hot topic: DNN compression)
- python engineer: model conversion and build a model zoo (Segmentation, Body Pose, Sound)
- python engineer: context adaptation product structuration (API rest) and server resource sharing
- hardware engineer: implement low level inference kernels (c/c++/ros) for several inference engines of our partners. Create the test benches for ST, NXP, Qualcom, GreenWaves, ...
</small>

Possibility to organize a happy hour if we get many candidates.

<small>contact: Lucas Fischer (lf@datakalab.com)</small>

---
# Internship Offers (2022-2023):
ISIR proposes 1 internship for research.

- emotion detection and face analysis: generation and manipulation of facial images for FER specialisation with few data.

This offer can lead to a PhD contract.

<small>contact: Kevin Bailly (kb@datakalab.com)</small>
---

class: middle, center

# Lab 1: here in 15min!