Image defogging system. A Retinex real-time image defogging method based on zynq is proposed. The brightness component V is defogged by Retinex algorithm in HSV color space; Using arm + FPGA Software and hardware cooperation, arm completes the algorithm control function, image color space conversion, logarithm and other simple operations; In FPGA, the parallel algorithm of Gaussian kernel function and two-dimensional image convolution is used to estimate the illuminance of ambient light. The experimental results show that the proposed method has the advantages of fast processing speed, miniaturization, embeddability, portability and low power consumption, and can meet the performance requirements of outdoor video system.
introduction
Due to the haze weather, the image quality obtained by the outdoor vision system will be seriously degraded, which will not only be blurred and reduce the contrast, but also have serious color offset and distortion, which will affect the stability and accuracy of the outdoor vision system. Therefore, it is very important to effectively and quickly remove fog from the degraded image caused by haze weather.
There are two kinds of common defogging methods: defogging method based on image restoration and defogging method based on image enhancement. The former establishes the physical model of defogging image from the reason of image degradation, and restores a clear defogging image according to the model. This kind of method has good defogging effect, but the algorithm complexity is large, so it is not easy to implement on the hardware platform. The defogging method based on image enhancement does not consider the cause and mechanism of image atomization, but selectively enhances the required details. Retinex algorithm is commonly used in this kind of defogging method [4]. Retinex algorithm can better preserve the edge and other details of the image. The processed image has the advantages of moderate brightness and high contrast. Most of the existing defogging algorithms are implemented on PC, and their portability and flexibility limit their application in outdoor video system.
Zynq is an all programmable system on chip of Xilinx system, which integrates dual core arm cortex-a9 and FPGA. It has a large number of logic units, digital signal processor (DSP) and other resources, and can meet the performance requirements of real-time processing, miniaturization, embeddability, portability and low power consumption of outdoor video system.
Therefore, this paper proposes a Retinex real-time image defogging method based on zynq, which uses the cooperation of arm + FPGA Software and hardware of zynq to improve the image defogging performance of outdoor video system. In order to reduce the complexity of the algorithm and color fidelity, the method performs Retinex defogging on the brightness component V in the HSV color space, linearly stretches the saturation S to expand the dynamic range, and keeps the hue H unchanged to avoid color distortion. In terms of implementation architecture, arm completes some simple calculations, such as logarithm, subtraction, linear stretching and color space conversion in Retinex algorithm, while the complex and time-consuming calculation of two-dimensional convolution of Gaussian kernel function and image is handed over to FPGA, and parallel algorithm is adopted to improve the real-time processing speed, The data interaction between arm and FPGA is undertaken by the on-chip bus axi4 (Advanced extensible interface 4) [5].
one
Traditional Retinex algorithm
Retinex theory holds that an image can be decomposed into incident component and reflection component, that is:
two
Algorithm optimization
In the traditional Retinex algorithm, the image r, G and B channels are processed respectively, which is easy to cause color distortion, and the amount of calculation in the hardware implementation is very large, which is difficult to meet the real-time requirements of the system. Therefore, this paper adopts the Retinex algorithm based on HSV color space. The foggy image is converted from RGB color space to HSV color space to obtain hue H, saturation S and brightness V with small correlation. Keep the H component of the image unchanged, linearly stretch the saturation component s, and only process the V component with single-scale Retinex algorithm, which can reduce the amount of calculation of the algorithm, It can also avoid the defect that the traditional Retinex algorithm is easy to cause color distortion. The algorithm flow chart is shown in Figure 1.
The specific implementation steps are as follows: select the appropriate σ Value and template size, and construct Gaussian kernel function according to equation (4); Converting foggy images from RGB color space to HSV color space; According to Retinex algorithm, the logarithm of luminance component V is taken to obtain V1; The component V is substituted into equation (3) and convoluted with the Gaussian kernel function, and then the logarithm is taken to obtain the illuminance estimate V2 of the ambient light; The V1 component and V2 component are subtracted to obtain the brightness component V3 processed by the defogging algorithm; Stretching the saturation component s to obtain S1; The hue component h, the stretched saturation component S1 and the brightness component V3 processed by Retinex algorithm are converted into RGB color space and output to complete the image defogging process.
three
Hardware platform implementation
The hardware platform is based on the development board of zynq-7010, and the design of image defogging system is completed by means of arm + FPGA [6]. In the image defogging system, image data is input as data; Logarithm module, color space conversion module, subtractor, linear stretching s module and display control are completed in arm; The convolution operation with large amount of calculation and long time is run in FPGA.
3.1 normalization of image data
In order to improve computing speed and reduce FPGA resource consumption, fixed-point numbers are used to save data and calculate. Experiments show that for the value range of HSV of image data [0,1], when the error is less than 4 × 10-3, there will be no obvious loss of image detail information. Therefore, by normalizing the value range [0,1] of image data with 0.003 906, the image data can be represented by 8-bit binary fixed-point decimal as shown in Figure 2. The decimal point is fixed to the left of the highest bit, the representation range is [0,0.996], and the minimum resolution is (0.00000001) 2 ≈ 0.003 906.
3.2 part of arm algorithm implementation 3.2.1 logarithm module
The Retinex algorithm needs to perform logarithmic operation on the foggy image I (x, y) and the incident component of the image (illuminance of ambient light) l (x, y). In order to reduce the quantization error and improve the real-time performance of operation in zynq system, the logarithm lookup table method based on 2 is used to realize the logarithm operation.
Since the image data I (x, y) and l (x, y) have been normalized to the 8-bit binary fixed-point decimal shown in Fig. 2, its value range ∈ [0, 0.996], the minimum resolution is 0.003 906, and there are 256 data values in total. Where the logarithm of 0 is - ∞, which cannot be quantified directly, so [0, 0.995] is divided into {0} ∪ [0.003, 9, 0.995][ The log field ∈ [- 8, - 0.007] corresponding to 0.003, 9, 0.995] is less than 4 × At 10-3, the numerical value can be represented by 16 bit signed binary fixed-point numbers with 1 bit symbol bit + 7 bit integer digit + 8 bit decimal digit; Log20 is approximated by the maximum negative number - 128 that can be represented by a 16 bit signed binary fixed-point number. Finally, the quantized 256 16 bit logarithms are stored in the lookup table LUT [].
X is used to uniformly represent the 8-bit binary fixed-point decimal of image data I (x, y) and l (x, y) that have been normalized, and Y is used to represent the 16 bit signed binary fixed-point number of logarithmic operation results, then the above logarithmic look-up table method can be expressed as:
Where the function int () represents the rounding of binary numbers.
3.2.2 linear tension s
In the process of defogging, with the enhancement of image brightness V, the saturation S will be relatively reduced. Therefore, in order to maintain the original brightness of the image color, it is also necessary to enhance the saturation S of the image. Here, the saturation S is stretched by three-stage linear stretching.
3.3 some FPGA algorithms realize the normalization of Gaussian kernel function
Experimental simulation shows that when σ= 40. The filter template is 101 × 101, the fog removal effect of Retinex algorithm is better. At this time, the value range of Gaussian kernel function is [2.085] × 10-5，9.947 × 10-5], the value is small. If directly quantized, the number of data bits is wide, and the resource consumption will be increased. Because the convolution operation is a linear operation, the Gaussian kernel function can be expanded first, and then the convolution result can be reduced by the same multiple without affecting the final calculation result. Here, the Gaussian kernel function is expanded by 27 times. The value range of the expanded Gaussian filter function is [0.002 7, 0.012 7], and the error is less than 2.56 × 10-4. There is no obvious loss of detail information in the defog image. Then use 2.44 × 10-4 the value range [0.002 7, 0.012 7] of Gaussian kernel function with 27 times normalized expansion is represented by 16 bit binary fixed-point decimals in total of 1 bit symbol bit + 3 bit integer digit + 12 bit decimal digit. As shown in Figure 3, the value range is [- 8,7], and the minimum resolution is (0.00000000000 1) 2 ≈ 2.44 × 10-4, which can meet the accuracy requirements. Finally, the value of the quantized Gaussian kernel function is stored as an initialization file in the form of lines and directly solidified into the block random access memory (Bram) of FPGA in the form of ROM.
3.3.2 convolution module
Convolution operation is composed of a large number of matrix multiplication and addition operations. Because its algorithm is in parallel structure, convolution operation is completed in the FPGA of the chip. The overall framework of the algorithm is shown in Figure 4.
Arm sends picture data and control information to the control register Ctrl of FPGA through axi4 bus_ REG and image data register row_ Reg, and read the current state of the control register and the convolution register result through axi4_ Results in reg.
The convolution operation top-level module consists of Gaussian kernel function ROM block Coe_ ROM, image block ram img_ Ram and computing module calc_ Module consists of three parts. The Gaussian kernel function used for convolution is stored in the Bram of FPGA and read in the form of Rom. the size of image block ram is the same as that of filter core, and the main control program is from COE respectively_ ROM and img_ The Gaussian kernel function and pixel data read from RAM are sent to the calculation module for calculation, and the calculation results are sent to the result register result_ reg。
Since the Gaussian filter core is 90 ° rotationally symmetrical and the Gaussian kernel function does not change in the program, in order to save storage space and reduce the complexity of logic control, only the upper part of the Gaussian kernel function is stored in the program, and the structure of the Gaussian kernel function is shown in Fig. 5.
In order to improve the calculation speed, the image data RAM block is saved by shift register. The image ram block slides from top to bottom and from left to right. The convoluted data is re prepared at the top of each column, and then the convolution block slides downward, moving one line downward for each calculated point. The shift register structure of the image block ram is shown in Fig. 6.
four
Experimental results and analysis
In the experiment, the hardware platform uses alinx7010, a black gold development board based on zynq-7010, and the FPGA clock is 250 MHz. The programming uses vivado 2016.3 provided by Xilinx company, and uses its own simulation software for simulation.
4.1 experimental results
In order to verify the performance of the proposed method, the performance of the proposed method is compared with the Retinex algorithm implemented on PC (core i5 6600k, 4.1 GHz dominant frequency, 16 GB memory, matlab 2013a). Both implementation methods are selected σ= 40, Gaussian template size is 101 × 101。
FPGA resource occupancy indicators include LUT, look-up table RAM (lutram), Bram, DSP, etc. Table 1 shows the four resource occupation tables of FPGA in the hardware part of zynq.
In the experiment, peak signal-to-noise ratio, information entropy and running time are used as three objective technical indexes to evaluate the effect of image defogging. Peak signal-to-noise ratio is the ratio of the maximum possible power of a signal to the destructive noise power affecting it. Information entropy is a technical index to measure the amount of information contained in the image and the richness of image details. The greater the entropy, the greater the amount of information contained in the image and the richer the image details [7].
Figure 7 shows the effect comparison diagram of three groups of image defogging experiments, and table 2 shows the comparison of the three performance indexes of the Retinex algorithm implemented on PC and the proposed method.
4.2 result analysis
It can be seen from table 1 that LUT occupies 76.31% of FPGA resources, and lutram, Bram and DSP occupy about 50% of FPGA resources, indicating that the whole system consumes less zynq-7010 resources, that is, the required hardware cost is low and can meet the needs of practical projects.
From the group A, B and C images in Fig. 7, it can be seen that the Retinex algorithm implemented by PC and the image defogging method proposed in this paper have obvious defogging effect.
It can be seen from table 2 that the peak signal-to-noise ratio processed by the two algorithms has reached more than 50 dB, which can effectively filter the noise caused by haze in the image and improve the definition of the image. The information entropy of the processed image is higher than that of the fogged image, which shows that the image detail information has been enhanced after defogging. Compared with the traditional Retinex defogging algorithm on PC, the method proposed in this paper achieves the same level in the indexes such as peak signal-to-noise ratio and information entropy, while the operation speed is increased by more than 28 times, and the real-time processing speed of more than 25 frames per second is achieved. The experimental results show that the implementation of image defogging on zynq platform has the advantages of fast operation speed, high portability and low power consumption, and can meet the practical requirements of outdoor video system., Read the full text, original title: [academic paper] Retinex real-time image defogging based on zynq
The source of the article: [micro signal: ChinaAET, WeChat official account: electronic technology application ChinaAET] welcome to add attention! Please indicate the source of the article“
Our other product:
