Due to the nonlinear effects produced by the actual defocusing projection system, which affect the accuracy of phase measurement, the phase error of binary fringe defocusing projection was studied. Based on the analysis of the current study status in the field, an expression for the intensity distribution of deformed fringe pattern signal in nonlinear systems is given, and the reasons for both high-order spectra components occurrence and their mixing with the fundamental frequency components, resulting in spectra overlapping, are analyzed. Defocus the projector to remove the higher-order harmonic components in the spectra domain and filter out one of the fundamental frequency components. An inverse Fourier transform was then performed on the spectra to obtain the expression of fringe intensity in the spatial domain. The continuous phase containing continuous signals was obtained using the phase-shift algorithm and phase unwrapping, and the expression for phase error after unwrapping in actual measurement systems was derived. The correct analysis of the basic principles has been verified through simulation and experiments. The simulation results indicate that the errors value obtained by the method mentioned in this paper are 34.51% for the binary fringe defocusing method, 44.83% Sampling method of Ref. [1], and 67.83% for method of Self-correction method of Ref. [10], respectively. The experiment results indicate that the phase recovered by using our method has good effects, and the corresponding phase error is relatively small.