This code is an accelerated open-source software for 2D optical flow estimation and mechanical strain. The code is based on D. Sun 1 method with some improvements.

LO09

Y. Li and S. Osher. A new median formula with applications to pde based denoising. Commun. Math, 7:741–753, 2009.

SRB10

D. Sun, S. Roth, and M. J. Black. Secrets of optical flow estimation and their principles. IEEE computer society conference on computer vision and pattern recognition., 17:2432–2439, 2010.

Requirements

There are two versions of the software: a CPU and a GPU version. To use the code, you will need Python 3 (3.6 or higher) with the following modules:

A. For the CPU version:

Note

You may also need Numba if Li and Osher filter 2 will be used

B. For the GPU version:

For the GPU version you first need an NVIDIA CUDA GPU with the Compute Capability 3.0 or larger. Beside the previous packages some additional ones will be needed to run the code on GPU

Note

The GPU version was tested using Cupy 9.2.0 and cuCIM API 21.10.01

Method

A. Energy:

Energy to minimize:

\[\begin{split}\begin{equation*} \begin{aligned} E(u,v,\hat{u},\hat{v}) =& \textcolor{black}{\sum_{i,j}{\Bigg\{ \rho_D( I_1(i,j)-I_2(i+u_{ij},j+v_{ij}) ) + \lambda[\rho_S(u_{ij}-u_{i+1,j})} }\\ &+\textcolor{black}{\rho_S(u_{ij}-u_{i,j+1})+\rho_S(v_{ij}-v_{i+1,j})+\rho_S(v_{ij}-v_{i,j+1})]\Bigg\} } \\ &‌+ \textcolor{blue}{\lambda_2(\mid\mid u-\hat{u} \mid\mid^2+\mid\mid v-\hat{v} \mid\mid^2)}\\ &+\textcolor{OliveGreen}{\sum_{i,j}{\sum_{(i',j')\in N_{i,j}}{\lambda_3(\mid{\hat{u}_{ij}-\hat{u} _{i'j'}\mid}+\mid{\hat{v}_{ij}-\hat{v}_{i'j'}}\mid)}}} \end{aligned} \end{equation*}\end{split}\]

Where:

  • \(\textcolor{black}{\text{Term in braces is the same as classical methods}}\)

  • \(\textcolor{blue}{\text{Encourages $(\hat{u}, \hat{v})$ and $(u,v)$ to be the same}}\)

  • \(\textcolor{OliveGreen}{\text{Smoothes $\hat{u}, \hat{v}$}}\)

Table 1 Parameters and their definition

Parameter

Definition

\(\hat{u}\)

The auxiliary horizontal flow fields

\(\hat{v}\)

The auxiliary vertical flow fields

\(N_{ij}\)

The neighbouring pixels of the pixel (i,j)

\(\lambda _2 \text{ and }\lambda _3\)

Scalar weights

\(\rho_D \text{ and }\rho_D\)

Penalties

B. Minimization

The previous objective will be optimized by alternately minimizing:

\[\begin{split}\begin{equation*} \begin{aligned} E_O(u,v) =&\sum_{i,j}\rho _D( I_1(i,j)-I_2(i+u_{ij},j+v_{i,j}) ) \\ &+\lambda[\rho_S(u_{ij}-u_{i+1,j})+\rho_S(u_{ij}-u_{i,j+1})+\rho_S(v_{ij}-v_{i+1,j}) \\ & +\rho_S(v_{ij}-v_{i,j+1}) ]+\lambda_2(\mid\mid u-\hat{u}\mid\mid^2+\mid\mid v-\hat{v} \mid\mid^2) \\ \end{aligned} \end{equation*}\end{split}\]

And

\[\begin{split}\begin{equation*} \begin{aligned} E_M(\hat{u},\hat{v})=& \lambda_2(\mid\mid u-\hat{u}\mid\mid ^2+\mid\mid v-\hat{v}\mid\mid ^2)\\ &+\sum_{i,j}\sum_{ (i',j') \in N_{i,j} } \lambda_3(\mid{\hat{u}_{ij}-\hat{u}_{i'j'}}\mid+\mid \hat{v}_{ij}-\hat{v}_{i'j'}\mid) \text{ } \\ \end{aligned} \end{equation*}\end{split}\]

\(E_O\) will be minimized first during the alterning optimization with \(\hat{u},\hat{v}\) fixed. Then with fixed \(u,v\) we minimize \(E_M\)

\(E_M\) will be optimized by applying Li and Osher’s median filter or approximately optimized by applying a simple median filter.

C. Summary of the method

  • Build a pyramid of the images.

  • Using the following derivation kernel: \(h=\frac{1}{12}[-1, 8, 0 ,-8 ,1]\)

  • Cancelling the derivatives at pixels where the movement is out of the edges

  • Compute the steps \(du,dv\) by solving a linear system using Preconditionned Minres.

  • Update \(u\) and \(v\)

  • Compute \(\hat{u},\hat{v}\) by Li and Osher or a simple median filter at each iteration