Male infertility (MI) is a complex multifactorial disease, and idiopathic infertility accounts for 30% of cases of MI. At present, the evidence for the effectiveness of empirical drugs is limited, and in vitro fertilization is costly and may increase the risk of birth defects and childhood cancers. Therefore, affected individuals may feel obliged to pursue natural remedies. Traditional Chinese medicine (TCM) may represent a useful option for infertile men. It has been demonstrated that TCM can regulate the hypothalamic-pituitary-testicular axis and boost the function of Sertoli cells and Leydig cells. TCM can also alleviate inflammation, prevent oxidative stress, reduce the DNA fragmentation index, and modulate the proliferation and apoptosis of germ cells. Furthermore, TCM can supply trace elements and vitamins, ameliorate the microcirculation of the testis, decrease the levels of serum anti-sperm antibody, and modify epigenetic markers. However, the evidence in favor of TCM is not compelling, which has hindered the development of TCM. This review attempts to elucidate the underlying therapeutic mechanisms of TCM. We also explore the advantages of TCM, differences between TCM and Western medicine, and problems in existing studies. Subsequently, we propose solutions to these problems and present perspectives for the future development of TCM.