Optical Flow Matlab/C++ Code
[Update Aug 1, 2011]. Recently I used successive over-relaxation (SOR) to replace conjugate gradient (CG) in solving the linear system and the code ran much faster!
I developed some C++ optical flow code that can be used in MATLAB during my thesis. I realized that it will be helpful to share this code to the research community. This code is purely for research/educational purposes. Please contact me if you want to use any part of the code for commercial purposes. You may contact me if you find that there is a bug in the code, but I am not responsible for compiling it for you. Moreover, please keep in mind that optical flow does not produce the expected results for many video sequences [4]. Please refer to human assisted motion annotation for obtaining more accurate motion.
When you refer to this code in your paper, please cite my doctoral thesis:
C. Liu. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral Thesis. Massachusetts Institute of Technology. May 2009.
Algorithm
The core of the algorithm is based on [2] and [3]. The major difference is that I used conjugate gradient for solving large linear systems instead of Gauss-Seidel or SOR. Why conjugate gradient? The solver becomes very simple and easy to code. If in every inner step we need to solve a large linear equation Ax=b, where x is the flow field, then we may find that matrix A can be decomposed to concatenations of filtering and weighting. So in a conjugate gradient solver, we never need to formally write down matrix A, but only need to write a function that consists of filtering and weighting to apply A to x.
In the appendix of my thesis, I also showed how to derive optical flow solver from the angle of iterative reweighted least square (IRLS) instead of Euler-Lagrange. IRLS and Euler-Lagrange end up with the same equations, but IRLS is much easier to understand and simpler to derive.
Although there can be multiple extentions to the optical flow as shown in Appendix A of my thesis, in this package only two-frame optical flow code is attached.
Examples
Several examples of applying the optical flow code in the package are shown below. These examplar images are also included in the package. Once you run the code, it should give you the same result. As mentioned in [4], there are two criteria for human to judge the quality of a flow field:
-
The brightness constancy assumption should be satisfieid, namely when you flip back and forth frame 1 and warped frame 2 (in column (c)), you shouldn't see any movement
-
The discontinuity of the flow field should agree with the object boundaries
So it's your judge whether the following flow fields are good or not. The ground-truth motion of these examples can be found here.
 |
 |
 |
 |
 |
 |
| (a) Flipping between image 1 and 2. | (b) Estimated flow field (visualized in the same way as [1], where hue indicates orientation and saturation indicates magnitude) | (c) Flipping between image 1 and warped image 2 according to the estimated flow field in (b) |
How to use?
The core of the package is written in C++ and mexed to dll to be called in Matlab. In the /OpticalFlow/mex folder, run the following
mex Coarse2FineTwoFrames.cpp GaussianPyramid.cpp OpticalFlow.cpp
You will obtain a dll file Coarse2FineTwoFrames.mexw64 (the extension can be different in Win32, Win64, MacOS and Linux systems). Copy this file to /OpticalFlow, and run demoflow.m to see a demo. Run help Coarse2FineTwoFrames for more details of setting up parameters.
Enjoy!
References
| [1] | S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. In Proc. IEEE International Conference on Computer Vision (ICCV), 2007. |
| [2] | T. Brox, A. Bruhn, N. Papenberg, and J.Weickert. High accuracy optical flow estimation based on a theory for warping. In European Conference on Computer Vision (ECCV), pages 25–36, 2004. |
| [3] | A. Bruhn, J.Weickert and C. Schn¨orr. Lucas/Kanade meets Horn/Schunk: combining local and global optical flow methods. International Journal of Computer Vision (IJCV), 61(3):211–231, 2005. |
| [4] | C. Liu, W. T. Freeman, E. H. Adelson and Y. Weiss. Human-assisted motion annotation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-8, 2008. |
Last update: Dec, 2009