Sea-keeping model tests of ships based on transient waves have been widely applied over the past several decades. In order to obtain response amplitude operators (RAOs) of a ship, most of the post-processing of the experimental data uses the fast Fourier transform (FFT) to obtain the wave spectrum and the corresponding response spectrum. However, for transient waves related model tests, FFT may produce larger errors due to its characteristics. Hilbert-Huang transform (HHT) is a newly developed signal analysis tool which is suitable for nonlinear and non-stationary data. The application of HHT to the post-processing of the experimental data of sea-keeping model tests of ships has not yet been investigated. In this study, the transient wave packets satisfying a Gaussian wave spectrum were generated in a large towing tank to conduct the sea-keeping model tests of a drilling ship under the condition of head waves, oblique waves and beam waves, respectively. Then the marginal Hilbert spectrum (MHS) in the framework of HHT is introduced to obtain the motion and the acceleration RAOs the drilling ship. In order to demonstrate the effectiveness of the approach, the results based on FFT and regular waves are also presented. It is found that in most cases, in comparison to that by means of FFT, the RAOs of the ship based on the transient Gaussian wave packets by means of MHS agree better with the results based on regular waves, especially for roll motion with significant nonlinear characteristics. Due to the advantages of HHT, the MHS approach employed in this study is expected to play a vital role in more sea-keeping related model tests of ships.