The output characteristics of sensor with two-circle coaxial optical fiber(TCCOF) are affected by the lubricating oil when the optical fiber sensor is adopted to measure the thickness of lubricating film. The effect of propagating medium on the maximum incidence angle of fiber is analyzed basing on the refraction law. The mathematical model of sensor in the lubricating oil environment is built taking into account the light loss and adopting the light intensity quasi-Gaussian distributing law. Meanwhile the output characteristics of sensor in the different propagating medium are gained by the simulating calculation of the model. Based on that the calibration experiment rig is built to verify the simulating output characteristics of sensor. The results indicate that the sensibility of TCCOF sensor are 0.691/mm in the air and 0.464/mm in the lubricating oil, respectively; the TCCOF sensor can eliminate the effect of changing optical power; the sensibility of TCCOF sensor increases with the increase of reflector curvature radius; the symmetrical structure of TCCOF sensor can decrease the effect of the different peripheral installing direction on the output of optical fiber sensor. Therefore, the TCCOF sensor must be calibrated in the corresponding lubricating oil and reflector environment.