Objective To measure the changes in total factor productivity (TFP) of tertiary hospitals in Shanghai and provide assessment indexes and decision making evidence for improving the performance of public hospitals. Methods Panel data (including 2 input and 2 output indexes) between 2003 and 2014 on 16 tertiary hospitals in Shanghai were collected and analyzed with Malmquist Index. Results The annual TFP growth rate in sample hospitals in 12 years was 7.2%, the annual technical progress growth rate was 6.4%, both technical efficiency and pure technical efficiency growth rates were 0.8%, and the scale efficiency remained unchanged. Conclusion The growth of TFP in tertiary hospitals in Shanghai is slow and the main contributor is technical progress. The growth rate of technical efficiency is very low. It is clear that the improvement of hospital performance should focus on enhancing the internal management, increasing the input-output efficiency of medical care resources, as well as integration of parametric and nonparametric methods for the comprehensive assessment of hospital performance.