Aiming at preventing stick-slip oscillations in drilling systems for oil and gas explorations, a reduced-order model is proposed to capture the nonlinear torsional dynamics of drilling operations. In this model, the drill string structure is simplified as a single-degree-of-freedom (DOF) system suffering from dry frictions at the drill bit, while the electromechanical boundary generated by the top drive system is modeled as another tunable DOF used for stick-slip suppression. To simplify and parameterize the problems, a normalized 2DOF system with negative damping and tunable parameters is deduced via nondimensionalization and linearization. Based on this system, stability criteria are obtained analytically in the five-dimensional parametric space. Stable regions and the optimized boundary parameters are found analytically. The results suggest that the system can be stabilized by an optimally tuned boundary when and only when the magnitude of the negative damping is no greater than . It also reveals that the stability deteriorates if the inertia on the top is huge and nonadjustable, which is the commonest scenario for commercial drilling rigs nowadays. Finally, applications of the tuned boundary in a typical drilling system for stick-slip mitigation are conducted and verified numerically. The results indicate that the control performance can be potentially enhanced by three to five times, via an additional virtual negative inertia generated by the top drive motor. This research provides an alternative approach to fully optimize the top boundary for curing stick-slip vibrations in drilling systems.