To improve the accuracy of the dynamic gesture detection, Gaussian mixture background modeling based on YCbCr color space is applied to the dynamic gesture recognition, and the effective gesture shadow elimination algorithm is proposed. First of all, the gesture image is cut out from video frame to be detected, and space ellipse fitting is developed in YCb'Cr' color. Oval color model is established statistically, and then dynamic gesture in the YCbCr color space through Gaussian mixture background modeling is detected. Original image is dotted product to get the gesture RGB image containing shadows. The shadows contained in the detected gestures image were eliminated by using ellopse color model, and finally we take gesture images together into a video sequence. Experimental results show that in the algorithm of dynamic gesture detection rate is 91.4% under a complex background, about 10% higher than that by the traditional methods. So it can meet the basic requirements of dynamic gesture detection, and has a high practical value.