A new algorithm, named the zipper layer method, has been developed to link multiblock meshes for groove-casing optimization applications. Numerical results for a turbomachinery rotor flow case are included to demonstrate the solution behavior across the zipper layer mesh. By using this new meshing methodology, the optimization of the casing groove geometries in relation to stall margin and efficiency of a transonic rotor is conducted. Six grooves are parameterized by their independent depths and a width to gap ratio. An advanced response surface method based on the Sobol design of experiment (DoE) and the Kriging response surface model (RSM) are used for the optimization. A leave-one-out cross-validation (LOOCV) method is used to calculate the quality of the response surface metric. The final optimized groove configuration is obtained through an optimization cycle using the Rolls-Royce SOPHY (SOFT-PADRAM-HYDRA) software (Shahpar, S., 2005, “SOPHY: An Integrated CFD Based Automatic Design Optimisation System,” Report No. ISABE-2005-1086), which not only improves the stall margin (SM) of the rotor but also maintains its peak efficiency.