Nonlinear vibration, a main factor affecting the dynamic stability, widely exists in the transmission system. In addition, geometric eccentricities caused by the manufacturing errors are inevitable in the gear transmission system, which may lead to the excessive nonlinear vibration. In order to suppress the nonlinear vibration under the excitation of the geometric eccentricities, a rigid–flexible gear pair consisting of the ring gear, the composite material, and the hub is proposed in this study. A dynamic model with nine degrees-of-freedom which considers geometric eccentricities is proposed to analyze the nonlinear dynamic characteristics of the rigid–flexible gear pair. Furthermore, the dynamic characteristics of the rigid–flexible gear pair and the rigid gear pair are compared within a wide range of operating conditions. The comparative analysis demonstrates that the rigid–flexible gear pair has better vibration suppression effect on the system.