Abstract: In order to solve the problem of “Egg-Chichen Absurdity” in the rate control model H.264, this paper presents an algorithm which can express the coding complexity of current macro-block using the average prediction residual. Firstly, a more precise bit allocation scheme is implemented according to the coding complexity of each macro-block, and then the quantization parameters of macro-block are calculated using the improved quadratic Rate-Distortion(R-D) prediction model and R-D optimization is achieved. Finally, the coefficients of the rate control model have been updated. This algorithm is implemented in JM8.6, and the experimental results show that the proposed method can improve 0.165 kb/s than JVT-H017 in the control accuracy, and improve 0.13 dB in PSNR on the average at the same rate. It can be concluded that the proposed method can control the rate more accurately and can obtain a higher quality subjective vision.