A new design strategy of gear system is proposed in this paper. Traditionally, development of gear systems undergoes as such. First gear structural parameters are determined. Then control of dynamic behaviours such as control of vibration is dealt with, where gear structural parameters cannot be changed any more. The new strategy suggests that during the design process of determining gear structure parameters, vibration control is taken into account. In order to see the effects of such a design strategy, an optimal design model is developed which contains two conflicting objectives, system vibration and system volume. The design variables include both discrete ones and continuous ones. Genetic algorithm, together with Pareto sets, is employed to solve this complex optimal problem. Through the simulation studies on selected cases, improvement on the system vibration control is shown with this new design strategy.