This paper addresses the optimal design of a novel nontraditional inerter-based dynamic vibration absorber (NTIDVA) installed on an undamped primary system of single degree-of-freedom under harmonic and transient excitations. Our NTIDVA is based on the traditional dynamic vibration absorber (TDVA) with the damper replaced by a grounded inerter-based mechanical network. Closed-form expressions of optimal parameters of NTIDVA are derived according to an extended version of fixed point theory developed in the literature and the stability maximization criterion. The transient response of the primary system is optimized when the coupled system becomes defective, namely having three pairs of coalesced conjugate poles, the proof of which is also spelt out in this paper. Moreover, the analogous relationship between NTIDVA and electromagnetic dynamic vibration absorber is highlighted, facilitating the practical implementation of the proposed absorber. Finally, numerical studies suggest that compared with TDVA, NTIDVA can decrease the peak vibration amplitude of the primary system and enlarge the frequency bandwidth of vibration suppression when optimized by the extended fixed point technique, while the stability maximization criterion shows an improved transient response in terms of larger modal damping ratio and accelerated attenuation rate.