Members of the silver-medal winning women's football team, dubbed the Nadeshiko after a pink mountain flower, were among the Olympians wearing red jackets and riding open-air buses through the main street of the glitzy Ginza shopping district.
All-around male gymnastics champion Kohei Uchimura showed off his gold medal with a big smile as he waved to the throngs lining the route.
The 38 medals won at the London games were the most ever by a Japanese Olympic team.
While only seven of those were gold, the athletes' performance was an encouragement to many in this country struggling to overcome years of weak economic growth and last year's devastating earthquake and tsunami that set off a nuclear crisis.
"They boosted our spirits," said Akiko Ina, who came to the parade with her 7-year-old daughter Kanare, with a Japanese flag emblazoned on one cheek and number 15 on the other, for her favorite female footballer Megumi Takase.
"I was glad I could see her," said Kanare, who watched the matches live, sometimes in the middle of the night in Japan.
The Japan Olympic Committee estimated the parade's crowd at 500,000 people.