Sent into bat by host captain Elton Chigumbura, the Indians relied on the openers - Vijay's 95-ball-72 and Rahane's 63 off 83 balls to lay a foundation with an opening stand 112 in 26 overs.
In-form Ambati Rayudu contributed 41 off 50 balls, hitting three boundaries, while Rahane found the fence seven time. Vijay struck a four and two sixes.
Manoj Tiwary (22) and Robin Uthappa (13) however missed yet another opportunity to impress the selectors and team management for a long term slot in the squad.
Leading 1-0 in the three-match series, India started well with the Rahane-Vijay combo sharing century opening stand.
There were partnerships worth 47 and 44 runs for the second and third wickets, with Rayudu involved in both, but the Indians could not acclerate the scoring.
Tiwary (22) failed to capitalise after getting a start, while the others such as Uthappa (13) and Stuart Binny (25) tried to prop up the innings towards the end with some brisk scoring.
Binny hit three boundaries in his 16-ball knock and it evebntually led to India scoring in excess of 270.
The Zimbabweans, to their credit, bowled to a good line and length and kept things tight for the Indian batsmen.
23-year-old medium pace bowler Neville Madziva, who was picked ahead of an injured Tinashe Panyangara for today's match, was the star performer for Zimbabwe, returning impressive figures of four for 47.
India have fielded the same team that won the opening ODI by four runs. Zimbabwe have made one change with Panyangara making way for Madziva.