The microstructure alteration generated in the high-speed machining of titanium alloy has significant influence on the performance, quality and service life of production. The prediction of grain size or phase distribution based on physics mechanism or the regression of experimental data have been reported in the process of static or quasi-static state. However, it is still a challenge to predict the phase transformation and grain growth process in machining accurately and effectively since it has characteristics of high strain, strain rate and temperature. In this paper, a novel FEM-based model involving with the microstructure alteration was introduced and implemented to predict finial grain size or phase result in the high-speed machining of Ti-6Al-4V alloys especially at the machined surface. The phase transformation process was proposed and discussed by considering tool wear and cryogenic condition at machined surface, while the microstructure results were displayed on the chip in the previous works. Firstly, the phase volume fraction and grain size were modelled by experimental data. Then the simulation based on the self-consistent method (SCM) was used to output strain and temperature distribution. Thirdly, the phase volume fraction and grain size expressions were transmitted into subroutine programs and the microstructure alteration process under the different cutting conditions were showed in the FE results. The simulation results of temperature, phase fraction and strain were compared against previous simulation or experiment results in published papers revealing good agreement. The proposed model was further to investigate the influence of tool wear and cutting temperature on machined surface. The results indicated that the tool wear increased heat at the flank face significantly resulting to β phase increasing and grain growth at machined surface and the cryogenic condition would lower temperature gradient as well as stress gradient contributing to reduce roughness and residual stress.