Fig.11a: We let the user choose which method is used to trigger an alarm. That is the "Level of Safety".
"High" means the naive area-ratio based method introduced in section VI of our paper. "Medium" means:
the approach of "learning" and threats-detection (or "working") alternatively. "Low" means
not only implementing "learning" and threats-detection (or "working") alternatively, but also only uses the headlamps
outside the bottom boundaries to trigger an alarm.
Fig.11b: The default setting is "High". That is, implementing the process of "learning" and "threats-detection" simultaneously.
The false alarm will be triggered.
Fig.12a: The demo when we choose "Medium". The "learning" and "working" processes are implemented alternatively.
During "learning" period, no alarm will be trigger. During "working" period, the learned convex hull will not shrink.
On the left-up corner of the smart phone, we use yellow word "learning" or red word "working" to
indicate which state is implemented. (This is used for debugging and testing.)
Fig.12b: During "working" period, the system uses the learned convex hull (without shrinking) for threats-detection.
In order to smooth the transition from "learning" period to "working" period, the first several updates in the "working" period will not be used to trigger an alarm.
Fig.13a: We shake the camera (by moving the tripod) on purpose. The system can learn the new (and correct) "normal traffic region" adaptively.
When we choose "Low", only the bottom boundaries will be used for triggering an alarm. This frame shows the result just before I move the tripod such that
the headlamps are moving upward suddenly (at 04:00 of video 2).
Fig.13b: The "Level of Safety" is chosen as low, thus, only the headlamps running outside the bottom boundaries of the
"normal traffic region" will trigger an alarm. Even we move the tripod such that
the headlamps are moving upward suddenly (at 04:00 of video 2) to run outside the
upper boundaries of the "normal traffic region" (comparing to Fig.13a), the alarm will not be triggered.
Fig.14a: During the "learning" period, a fixed convex-hull (with green boundaries) is used for danger detection.
This frame is at 8:55 in video(result on Samsung).
Fig.14b: The threat is detected and an alarm is triggered, even if
it happens during the learning period. This frame is at 8:57 in video(result on Samsung).
Fig.14c: During the "learning" period, a fixed convex-hull (with green boundaries) is used for danger detection.
This frame is at 16:26 in result on Nexus ("Middle").
Fig.14d: The threat, i.e. a car being closer to the road than other cars, is detected and an alarm is triggered, even if
it happens during the learning period. This frame is at 16:27 in result on Nexus ("Middle").
The video video(result on Nexus 6p) shows the false-alarm-rate (FAR) test of the result on Huawei Nexus 6P.
The FAR of naive "updating-and-shrinking" approach, i.e. with "Level of Safety" as "High", is pretty high. That is, one false alarm per 4 or 5 passing cars. By using the
method of implementing "learning" and "working" periods alternatively, i.e. with "Level of Safety" as "Middle", the FAR reduces greatly. The threat detection process becomes pretty stable. The false alarms caused
by shrinking of the convex-hull only happen in the transition between the "learning" and "working" periods. In the testing experiments, we set the ¡°learning¡± period as 100
updates and "working" period as 500 updates. In practice, the "working" period can be set much longer.
Then the false alarm rate can be reduced further more. (The false alarms caused by other reasons, e.g. motion of roadlamps caused by small shake of the tripod, are not taken into account
for the comparison of FAR corresponding to different approaches.)
Here is another testing experiment running on Huawei Nexus 6p.
The video result on Nexus ("High") shows the result of the naive "updating-and-shrinking" approach,
i.e. with "Level of Safety" as "High". The FAR is pretty high (i.e. one false alarm per several cars).
The video result on Nexus ("Middle") shows the result of implementing "learning" and "working" periods alternatively, i.e. with "Level of Safety" as "Middle".
Still, the FAR reduces greatly. The threat detection process becomes much more stable. Moreover, a potential threat, i.e. a car being closer to the road than other cars, during the "learning" period
is detected (See Fig. 14c and Fig. 14 d). (Note that the vibration on android watch is a little earlier (about half a second in our test)
than the display of alarm signal on both android phone and android watch.)
The obvious threats as shown in the demonstration (and in Fig. 9 and Fig. 14), i.e. a car turning and approaching the camera suddenly, are detected all the time, because
1). the headlamps of the car which is turning and approaching are large enough to be detected every time,
and 2). the convex-hull will also expanded suddenly in this case. Here are more results of testing the reliability of the system:
1. MoreTesting1, MoreTesting2 and MoreTesting3.
(Note that the vibration on the android watch is a little earlier (about half a second in our test)
than the display of alarm signal on both android phone and android watch.)
In order to make the interface looks neat, we don't show the fixed convex hull used for danger detection during
the "learning" period, i.e. the convex hull with green boundaries, in our final product. The danger detection during
the "learning" period is running on the "background" of the system.
Our attempt of stabilizing the threats-detection process is inspired by the good suggestion from the reviewer of our paper, that is, intelligent classification of potential threat should be used with the learned "normal traffic region"
in order to stabilize the threats-detection process. The approaches we have tried can be thought as the "naive classification of potential threats". For instance, is the threat (headlamps outside the convex hull)
dangerous with high probability (i.e. outside the upper boundaries of the convex hull)?
We should also mention that our naive methods are only the first-step attempt
of finding a good balance between reducing false alarm and avoiding missing real threats
(without the information about where the police officer is).
This is still an open problem and can be improved.
Complementary videos
- The following video shows the text of the basic modules of the system. They are all running on the smart phone on real time
- Demonstration of how to use the system:
Detection and tracking.
The experiment is running on the smart phone (Samsung Note 3).
- Demonstration of updating the convex hull to match the "normal traffic region" (alpha = 0.9).
The experiment is running on the smart phone (Nexus 6).
- Demonstration of triggering the alarm: demo1 and demo2. For android system, basically the developer need do
almost nothing the send a signal from the smart phone to android wear. We use ¡°LG G Watch R W110¡± as the receiver, which will generate vibration when
the alarm is triggered. We can design the pattern of the vibration
Fig.15a: In the default mode, the algorithm tracks the blobs and learns the "normal traffic region". If the tracks move outside
the "normal traffic region", the alarm signal will be sent, and trigger the vibration on an Android watch. This image is a randomly
chosen frame from the video demo1.
First several seconds (e.g. first 20 updates of the convex hull) are used to
"learn" the "normal traffic region".
Fig.15b: We can also set a "working region" in case the police officer is standing outside the police car.
Now, if there is intersection between the "working region" and "normal traffic region",
the alarm signal will be sent, and trigger the vibration on an Android watch. This image is a randomly
chosen frame from the video demo2.
Fig.15c: The real environment used for experiments of demo1 and
demo2. Although the lamps are not easy to be detected in such environment,
by setting low ISO (e.g. 100), then we can get good performance of lamps-detection (see Fig.15a and Fig.15b).
- For more fun, we create another two videos: video_demo1
and video_demo2.
My friend Dennis Porche helps me record the experiments result by another camera when I running the algorithm on the smart phone (Nexus 6P).
Fig.16a shows one randomly chosen frame from video video_demo1,
and Fig.16b shows one randomly chosen frame from video video_demo2.
Fig.16a: One randomly chosen frame from the video video_demo1.
Fig.16b: One randomly chosen frame from the video video_demo2.
- The Bluetooth communication works pretty well when the distance is up to more than 10 meters. My friend Frank Chen help me
do the following experiments. Frank Chen raises his hand when he feels the vibration from the Android Watch.
See the videos ReactionTest1
and ReactionTest2. (Note that there is extra reaction time needed for noticing the vibration on android watch,
and another extra time for the person to raise his hand when he feels the vibration on the watch. Thus, although the warning signal is sent promptly,
the person will raise his hand a little bit later (about half a second) after he feels the vibration.)
The vibration on the Android Watch is a little bit earlier than the display of warning information on both smart phone and
on the Android Watch.
Fig.17: My friend Frank Chen raises his hand when he feels the vibration from the Android Watch on his wrist. This picture is one frame of ReactionTest1.
- Demonstration of the basic idea of implementing "learning" and "working" periods
alternatively. See the video two_periods.
(Note that the vibration on the android watch is a little bit earlier than the alarm information
displayed on both smart phone and android watch. Thus, there is actually no noticeable delay.)
Fig.18a: During the "working" period, the learned convex hull doesn¡¯t shrink.
This image is the 23-sec frame of the video two_periods
Fig.18b: During the "learning" period, a fixed convex hull (i.e. the initial one in the "learning period")
is used for danger detection. The fixed convex hull for danger detection has green boundaries.
This image is the 25-sec frame of the video two_periods
Fig.18c: During the "learning" period, when some objects move outside the convex hull (with green boundaries),
an alarm will be triggered. This image is the 26-sec frame of the video two_periods (Note that the vibration on the android watch is a little bit earlier than the alarm information
displayed on both smart phone and android watch.)
- In order to make the interface looks neat, we don't show the fixed convex hull used for danger detection during
the "learning" period, i.e. the convex hull with green boundaries, in our final product. The danger detection during
the "learning" period is running on the "background" of the system.
- Analysis of the processing time per frame.
The most time-consuming module is ``objects-detection'' (about tens of milli-sec.). The time of
tracking (about 1 milli-sec) and convex-hull based opinion (1 to 2 milli-sec) is relatively small.
Fig.19a: The piece of code involving analyzing the processing time for each frame. "onInitialData()" generates the input data
(i.e. a down-sampled binary-image matrix) to the alerting system. "Region.regionGroup()" implements objects-detection discussed in section III of our paper.
"onTracking_new()" implements the multi-objects tracking introduced in section IV ofour paper. "normalTraffiRegion()" implements convex-hull
based intelligent identification of potential treat (discussed in section V of our paper).
Fig.19b: Testing results of processing time per frame. The corresponding code is in Fig. 19a.
Fig.19c: The piece of code involving analyzing the time used to send notification by android phone.
Note that the time of flight is not included. However, the size of the alarm signal is small (only hundreds of bytes).
Note that the data rate by blue tooth communication is more than 1M/s in general.
Fig.19d: Testing results of processing time for sending notification by android phone. Basically it's about 10 Milli-sec.
The size of the alarm signal is only hundreds of bytes. The corresponding code is in Fig. 19c.
Summary and Further Work
- For many Android smart phone, you can adjust the parameters of the camera. For Nexus 5, 6 and 6p, you can even
use the new Class "Camera2" to set these parameters in your JAVA code. Thus, we can set low ISO and short shutter time for obvious contrast between
headlamps and backgrounds (see Fig. 2).
- However, for some very unexpensive Android smart phone. You can not set these parameters.
Moreover, if the videos are taken when the smart phone is handheld, rather than mounted on a tripod.
It will contain sever handshaking.
- We should mention that in our project, we use the smart phone whose camera's parameters can be adjusted.
And the smart phone is fixed (mounted on a tripod or police car).
- Our work is just the beginning of this alert and divert method of
saving lives of roadside personnel. It's not perfect. There is still a lot of work to do.
- Our work is only suitable to the simplest environment
(in which headlamp detection is relatively easy). For more advanceed cases, more complicated detecting and tracking methods are needed.
Videos in special cases for experiments
