Deep Learning Lectures

class: center, middle

# Nerfs and Gaussian Splatting

Edouard Yvinec

.affiliations[
  ![Epita](images/Epita.png)
]

---

3D scene reconstruction

NeRF: Neural Radiance Field

Gaussian Splatting: 3D Gaussian Splatting for Real-Time Radiance Field Rendering

---
class: middle, center

---

class: middle, center

---

We need a function that gives the value of the pixel from a 3d beam.

$$ \int_0^1 \sigma(b(t))C(b(t)) dt $$

where $\sigma$ is the opacity function, $b$ is a parametrization of the beam and $C$ is the color function.

$$ \int_0^1 \sigma(b(t))C(b(t)) \times e^{-\int_0^t \sigma(b(s)) ds } dt $$

---

.center[
<iframe width="560" height="315" src="https://www.youtube.com/embed/JuH79E8rdKc?si=Fv-K7WRH4t5v0IJi" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
]

---

In summary,

1.  We work on beams (not images)
2.  One model per-scene
3.  Introduce a mathematical trick to represent the opacity.

---
class: middle, center

3D Gaussian Splatting for Real-Time Radiance Field Rendering

First, Gaussian Splatting leverages structure from motion similarly to NeRFs

---

Each data-point is replaced by a 3D Gaussian. A 3D Gaussian is defined by
--

$$
G:x\mapsto e^{-\frac{1}{2} x^T \Sigma x}
$$

For training, we need to be able to project our 3D gaussians to 2D representations
--
$$
\Sigma' = JW\SigmaW^TJ^T
$$
where W is the viewing matrix and J the Jacobian of the affine transformation which approximates the projective transformation. However the covariance matrix must be positive semi-definite.

--
To achieve this, the covariance is derived from a scaling matrix and a rotation matrix
$$
\Sigma = RSS^TR^T
$$

---
The positions, opacity, colors and covariance are optimized through gradient descent. After every 100 optimization steps, the density of the gaussian is controlled and updated.

--
The gaussians to update (densify), the idea is to tackle trhee specific cases:

1.  few Gaussians in large areas (under-construction)
2.  large Gaussians (over-construction)
3.  remove transparent Gaussians

---

.center[
<iframe width="560" height="315" src="https://www.youtube.com/embed/T_kXY43VZnk?si=HduRWl5mJpqB6JkS" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
]

---
class: middle, center
# Wishing you the best and see you at the project defense