Complete resection of submucosal invasive colorectal cancer (SICC) showing favorable histology is regarded as curative. We report on two cases of SICC showing recurrence within 5 years despite complete resection. The first patient was a 68-year-old woman with well differentiated rectal adenocarcinoma invading the superficial submucosa, which recurred after 4.7 years. The second patient was a 53-year-old man with pT1N0 moderately differentiated colonic adenocarcinoma. He developed widespread tumor recurrence after 3.9 years. Retrospective pathologic review of the original tumors showed multiple foci of tumor budding at the invasive front. Immunohistochemical staining for D2-40 of deeper levels of the paraffin blocks showed rare foci of small lymphatic invasion. Tumor budding at the invasive front may be an important indicator for SICC aggressiveness or may reflect early lymphatic invasion. More aggressive pathologic examination and follow-up is required for patients with SICC showing tumor budding, even in the absence of unfavorable histologic findings.