REAL-TIME LANE DETECTION ANALYSIS AT NIGHT

The Lane driving assistance is the need of the hour as the number of people using vehicles has increased. This may increase the chances of accidents. If we consider driving at night time, smart assistance for vehicles is even more important. We have worked on the lane detection, where we have implemented our algorithm for four different scenarios. We have used Adaptive Gamma Correction on dark frames and Inverse Perspective Mapping (IPM) is used with camera calibration and Kalman Filter (KF) plays an important role in predicting the lanes in the dark environment. The experiment conducted shows that our algorithm is robust to use in real-time night conditions

The Lane driving assistance is the need of the hour as the number of people using vehicles has increased. This may increase the chances of accidents. If we consider driving at night time, smart assistance for vehicles is even more important. We have worked on the lane detection, where we have implemented our algorithm for four different scenarios. We have used Adaptive Gamma Correction on dark frames and Inverse Perspective Mapping (IPM) is used with camera calibration and Kalman Filter (KF) plays an important role in predicting the lanes in the dark environment. The experiment conducted shows that our algorithm is robust to use in real-time night conditions. CopyRight, IJAR, 2019. All rights reserved.

…………………………………………………………………………………………………….... Introduction:-
Vision-based lane-detection methods provide low-cost density information about roads for autonomous vehicles [1]. Intelligent vehicle technologies have strongly developed in recent years because the number of vehicle accidents has increased annually. These technologies utilize sensors such as Lidar, Radar, and vision sensors. In which, Lidar and radar are used for obstacle detection, unique vision sensors are used for lane detection and vehicle detection. Detecting lane markings and vehicles enables vehicles to evade collisions and support a warning system. Moreover, today vision sensor is cheaper, smaller, and higher quality than before. For these reasons, lane detection and vehicle detection using a vision sensor is an interesting topic for researchers and engineers [2]. The next promising key issue of automobile development is a self-driving technique. One of the challenges for intelligent self-driving includes a lane-detecting and lane-keeping capability for advanced driver assistance systems [3].
The challenging task in developing autonomous vehicles is road followers. These techniques include road detection, lane detection, obstacle detection, and pedestrian detection. The accidents happen due to curvy lanes, multiple lanes, and lanes with a lot of discontinuity and noises [ 4]. Lane detection is crucial to Lateral Vehicle Guidance(LVG) and Lane Departure Warning(LDW) which can eliminate many avoidable accidents. Due to the great variety of road types (urban, suburban, rural and campus roads), weather conditions (sunny, cloudy, rainy, snowy and foggy) and illumination changes (shadow, glare, and darkness), it is not easy to develop an efficient lane detection algorithm that always works. The Hough Transform is commonly used in lane tracking systems as it is very powerful because it can detect road boundaries successfully even in an extremely snowy environment. [5] [6]. There has been active research going on the lane detection and a wide variety of algorithms of various representations, detection, and tracking techniques, and modalities have been proposed. [7] ISSN: 2320-5407 Int. J. Adv. Res. 7(10), 191-200 192 Lane detection at night is a very important aspect of Intelligent vehicles. As most of the accidents happen during night time than day time, due to impaired vision of the driver, etc [8]. We wanted to use the standard camera of mobile as it is readily available and everyone has android phones. The real-time video is streamed on the local server. The proposed approach used here is to get a robust lane prediction at night using a Kalman filter. The Kalman filter with Hough Transform would help in Tracking, Predicting and Updating of the lanes.

Figure 1: Camera Projection
The camera is placed on the dashboard inside the car, which gives better visibility and projection on the road for lane detection as special attention has to be taken for the night scenario. The projection model is constructed in such a way that the 3D point that enters the camera converts it into a 2D geometric model which is the image coordinate system having image projection(u,v). The camera perception is the most important part of this project where the lane markings in the real-world plane will intersect at a vanishing point in the image plane. In the diagram, Xc, Yc, Zc is the camera plane. The main interest of our project is to track the road lanes which has to be transformed into an image coordinate through the projection matrix. The camera calibration is an important part as it does not perceive as our human eye does. So, the camera matrix calibration has to be done to remove the distortions and to get an undistorted image, the difference might not be visible but it makes a difference.

Figure 2: Camera Alignment
We are interested in extracting the road plane data and eliminate all the redundant data other than the correct lane. So, the camera has to be adjusted with the proper angle (ᶿ) with respect to the road and, the height(h) would be taken from the plane of the dashboard of the vehicle with respect to the flat road. The desired lane would be termed as ROI (Region Of Interest). More about the ROI will be explained further in the ROI section.

Block Diagram:-
The project focuses on the lane detection of the road at night under different scenarios. Detection of lanes at night is very challenging due to various reasons such as high noise, little to no visibility of lane markers, poor edge detection, shadow reflections, and disturbances under the powerful streetlight and most importantly heavy headlight glare from the opposite vehicles in a two-way lane. So, to overcome such problems, we have worked on the algorithm that could eliminate most of these problems efficiently. It is important to note that all the work is done 193 using a standard full HD camera. For proper localization of vehicles in between the ROI, it is important to detect lane boundaries efficiently.

Pre-processing stage:
When working on the night scenes, it is important to think about the image that would be captured will be quite dark which might neglect most of the details and might hinder the desired results. So, the pre-processing stage is most important as it is the first stage of image processing. Its sole purpose is to boost the image quality and process it to increase its usability for the desired work. First, we have used the Adaptive gamma correction algorithm for the night image and then image smoothing is done using the median blur (best for our dataset). The color thresholding or segmentation does not work well in case of discontinuity of lanes, shadows, etc. So, we have gone with the edgebased segmentation using Canny edge-based segmentation which works for most of the cases.

Adaptive gamma correction
Gamma correction plays an important role when it comes to getting details and clarity on a dark image. The digital camera receives twice the number of photons which is linear in relation. But human eyes work differently and have a non-linear relationship with brightness. So, human eyes are more sensitive to changes in dark tones than bright ones. The Gamma correction works on the principle of Power Law Transform. We tried working on HSV color space as well but gamma correction gave better results for night condition. We have used it to get the highest visibility of the lane boundary. We have worked on the adaptive gamma correction which will be explained further in detail.

Region of Interest(ROI)
This is one of the mandatory parts in the lane detection algorithm as it crops out the unwanted region of the image or the frame of the video and focuses on the desired region, which eventually increases the computational speed and reduces the complexities. In our case, the camera is placed inside the car and the camera projection is in such a way that captures the surrounding with the lane. So, we crop the region at the top and sides of the road. The angle(ᶿ) of 194 the camera is adjusted in such a way that it is as small as possible to get more concentration towards the road. The height from the dashboard of the car to the road surface would remain the same assuming the road to be flat. The blank matrix is defined that matches the height and width of the image and the image is returned where the masked pixel match. In this way, we get rid of unwanted objects. Our main focus is to use the projective and perception model, where the camera geometry will play an important role in capturing accurate video frames.
The camera is fixed on the center of the dashboard with no yaw movements. So, we are not taking into account these values. So, we have calculated the height(h) and angle(ᶿ) for the best camera projection. The measured angle we got is ᶿ=67 degree with respect to the dashboard and the measured height from the surface of the road would be the same throughout the project assuming the flat road surface, h=1.22m measured using Chevrolet Spark used in our experiment.

Inverse Perspective Mapping(IPM)
After the ROI, which was based on the perspective projection, IPM is deployed on the ROI. It helps to remove the perspective effect from the acquired image, remapping it into a new 2D domain in which the information is homogeneously distributed among all pixels. The camera geometry is considered while working on IPM transformation. It helps us to get warped bird-eye-view of the target road, as well as the parallel lanes. This helps a lot in the night scene where road lanes are not clearly visible. Where it will refine the image and reduces the noise and will help to get more area of lane markers. It also helps to remove the far sight in the image which might create a lot of disturbance.

Detection Stage:
The stage after the pre-processing stage is the detection stage, where edge-based detection is done in the region of interest (ROI). The Canny edge detector is used for detecting edges of the image. The modified Hough transform similar to probabilistic Hough Transform is applied on the edged image and it helps in detecting the road lane. A line can be represented in the following forms, Where r is the perpendicular distance from the origin to the line.

Predicting and Updating lane lines:
Lane Marker's following is difficult at night time. We have incorporated Kalman Filter for this purpose.
[9]The Kalman filter is a computationally efficient, recursive, discrete, linear filter. It can give estimates of past, present and future states of a system even when the underlying model is imprecise or unknown. The advantage of using this algorithm is that it decreases the false detections and helps to predict future lane marking in the video frame. The hough transform that is used to detect the borders of the road can be tracked efficiently by the Kalman filter. It would use the hough parameters( r, ᶿ) and its derivative to implement it in real-time. It would then predict the future lanes based on the previous lane lines and would update that lane line in the state matrix. The approach used is the recursive Kalman algorithm, which has been used in this project.

Lane detection algorithm applied on night scene:-
The following three features have helped to get robust real-time lane detection at night. These fundaments, when implemented on the lane detection, will improve the accuracy of lane detection.

Auto gamma correction
The gamma correction gave us the best results out of all the image correction as gamma correction works well on dark images. When using it in the real-time video, the gamma correction value might fix at one constant value which might not give good results. So, we have changed the values based on the darkness of the image. This would help to show more details on the image. 196 The input image that will be provided, each of the color channel will be transformed to give a brighter output color image. The camera follows a linear relationship with brightness while our eyes perceive Non-linearly. So, the input signal from the camera will be transformed by the gamma transform, then the correction is done to give a gamma calibrated image. This helps the gamma encoding to be done properly. Otherwise, the image would become too dark than desired or too bright.

IPM( Inverse Perspective Mapping)
The Inverse Perspective Mapping is used in the preprocessing stage after ROI, this helps in getting the bird's eye view for road lines. Another advantage of IPM is it helps to remove the noise. The far-sighted view is removed and helps to eliminate the disturbance from the headlights of the car and the surrounding noises. The ROI points are taken and the inverse perspective mapping is done in that area, which improves the accuracy of lane detection. This is a big advantage at night scenes as it increases the probability of detecting the lane markers. The camera projection parameters are important for best lane detection.
[10] The pitch angle(ᶿ) is measured and is used. The angle should be in the range of 35.5 degrees to 90 degrees. The yaw angle is not considered and the camera is placed at the middle center of the car. The calibration of the camera is done and hence we obtain the focal length(fx, fy) and optical center(cx,cy). The camera parameter matrix is denoted by C, rotation matrix by R and Translation matrix by T: In our case, we have used 'findHomography' to do a perspective transformation between two planes: source points and destination points. The homography helps in getting a panoramic view. It helps to remove certain noises and disturbances. Then canny edge is used as it has high computational speed and is the most used operator for lane detection. The segmentation is done on the IPM image. We found that edge-based segmentation is better than the color segmentation when there are shadow/ illumination disturbances and lane discontinuities. The image is warped back to normal after that Hough transform is applied.

Predicting and Updating (Kalman filter)
The predicting and update of lanes is done by using Kalman Filter(KF). The tracking of lanes is done with the help of Hough Transform(HT). The Kalman filter is used for different scenarios at night. We have seen successful detection of lane lines in all of the four scenarios. In the prediction step, state variables are initially estimated by the Kalman filter, which also initializes the process noise, priori estimation error. Meanwhile, the system keeps monitoring measurement information and feeding measurement matrix together with measurement noise into the updating step.
At the updating step, results of prediction need to be updated based on the computable weight of estimation results and measurement results (by utilizing the innovation which indicates the certainty). Trust from the system depends on the certainty, which will recursively influence the next instances. [11]  Update the detected edge points (measurements) and the predicted state vector is used in the updating equation of the KF to estimate the new model parameters.
The Kalman filter helps in removing uncertainties in our estimation. We obtain the measurements of the state matrix from the captured frame and update the predictions based on the previous estimate. The Kalman gain is an important element, which is the regulator between our estimate and measurement.

Experimental Results:-
The experiment is carried out for four different lighting conditions at night to gain exposure to the real-time working of the Kalman filter. The image shown below shows the order in which the algorithm is applied.
Scenario1: This is a general well lit condition at night. Here the lanes were visible but were reflective. The gamma correction value is 0.5306. There are no vehicles in this scene and the detection of quite accurate.
Scenario2: There is a headlight flash of a vehicle in the other lane on a two-way road. The brightness was good enough and the gamma correction value is 0.522. This one was tested for the accuracy of canny edge detection under such disturbance.
Scenario3: This is the condition is in the traffic when there is a lot of lights and disturbances. The gamma-corrected value is 0.5155. The IPM works well in this scenario, keeps the lanes and then the edge detection is done on it which gives the desired result.
Scenario4: This is the condition when there is complete darkness. There is an absence of street lights. The gammacorrected value is 0.498. In this case, we wanted to test the real working of the Kalman Filter. It predicts the lane lines based on the previous frame lane lines and thus the lane line is tracked in the current frame even when edge detection fails to detect lanes to some extent.
The proposed system uses Intel i5 8 th generation-8250U CPU clocked @1.6GHz with 8GB DDR4X RAM. The primary full HD camera of Samsung S8 of aperture f1.7, 12MP and pixel size of 1.4micro meter is used and is placed on the dashboard. It is used as an IP webcam and the video is streamed on the local server. The programming is done using python and OpenCV in Jupyter notebook and Spyder IDE.

Conclusion:-
In this paper, the lane detection at different night scenarios is presented by using key algorithms/features such as Kalman Filter(KF), Modified Hough Transform (MHT), Adaptive Gamma Correction (AGC) and Inverse Perspective Mapping(IPM). The experiment is carried out in real-time and analysis is made. Firstly, due to dark frames of video AGC is done to extract lane details and then IPM is done on that frame to get the bird-eye-view and 200 remove noise. The edge segmentation is done and then the frame is warped back to normal. Secondly, MHT with KF is applied to get the best accuracy of lane detection even under the absence of street light (dark image). Finally, the above algorithm is overlayed on the real-time video stream to get lane detection. The experimental results show that the method is robust and efficient and can meet real-time requirements.

Future work:
We are working on the curved lane detection at night using a curve stitching algorithm and Extended Kalman Filter. It can be very challenging to work on the curved road at night using a standard camera. Using different sensors such as depth sensors, LiDAR can even help to deal with complex environments such as during rain, snowfall, etc.