High-pressure multistage pumps and their coupled piping systems, typically used in the process and power generation industry, can experience dangerous system-level instabilities. This can occur at flow coefficients well away from the surge limit and in the absence of cavitation. Such a pumping system and a related new kind of instability are the focus of this paper. A system-wide instability was observed at 0.05 times rotor frequency for flow coefficients near maximum head rise but at negative slope, thus on the stable side of the head rise characteristic. A previous study based on system-level experiments concluded that this instability differs from classical surge, cavitation surge, rotating stall, and rotating cavitation, but the underlying mechanism and necessary flow conditions remain unknown. This paper investigates the root cause of the system-wide pump instability, employing a systematic analysis of the impact of geometry changes on pump stability and performance. It is found that the upstream influence of the unsteady flow separation in the return channel leads to a time-varying incidence angle change on the volute tongue which causes periodic ingestion of low-stagnation pressure fluid into the diffuser passages. This sets up a limit cycle, promoting the system-wide instability. With the instability mechanism determined, the pump is redesigned to remove the flow separation while maintaining performance at design conditions. Unsteady numerical simulations demonstrate improved efficiency and pressure recovery at low flow coefficients. A time accurate calculation also indicates stable operation at all relevant flow conditions. The paper resolves a long-standing pump stability problem and provides design guidelines for reliable and improved performance, important to the chemical processing and power generation industry.