At present, the mean value of the meshing stiffness and the gear backlash is a fixed value in the nonlinear dynamic model. In this study, wear is considered in the model of the gear backlash and time-varying stiffness. With the increase of the operating time, the meshing stiffness decreases and the gear backlash increases. A six degrees-of-freedom nonlinear dynamic model of a new rigid-flexible gear pair is established with time-varying stiffness and time-varying gear backlash. The dynamic behaviors of the gear transmission system are studied through bifurcation diagrams with the operating time as control parameters. Then, the dynamic characteristics of the gear transmission system are analyzed using excitation frequency as control parameters at four operating time points. The bifurcation diagrams, Poincaré maps, fast Fourier transform (FFT) spectra, phase diagrams, and time series are used to investigate the state of motion. The results can provide a reference for the gear transmission system with wear.