In the optical communication, people use the optical fibers to achieve the high bit-rate data transmission. In this paper, the AB system for the ultra-short pulses in a nonlinear optical fiber is investigated via the Lie symmetry analysis. Lie symmetries and symmetry reductions are derived via the Lie algorithm. Periodic- and solitary-wave solutions are obtained via the qualitative consideration. For the magnitude of the electric field in the optical fiber and the function associated with the occupation number which gives a measure of the atomic inversion in the nonlinear optical fiber, we can adjust the amplitudes, widths, and velocities of the solitary waves via the Lie symmetry transformations. The results would help the engineers select the ultra-short pulses in the optical communication.