The military had stepped up attacks and airstrikes against al-Qaeda in the mountainous area of al-Rahha in the southern province of Lahj, a strategically important region that links the south with Yemen's northern cities.
The offensive followed two surprise attacks by militants on Yemeni army bases in the area.
The military officials told The Associated Press that the government forces are trying to reclaim key cities in Aden and Abyan provinces in the south, overrun by al-Qaeda.
The officials spoke on condition of anonymity because they were not authorised to speak to the media.
Al-Qaeda-linked militants have taken advantage of a year of internal political turmoil and security vacuum in Yemen to expand their gains in the country's south.
The militants have seized several towns and cities and entire swaths of land, and the military's campaign has so far not managed to retake those areas.
Al-Qaeda's branch in Yemen, known as al-Qaeda in the Arabian Peninsula, is one of the movement's most dangerous offshoots.
Yemen's uprising, inspired by Arab Spring revolts elsewhere, forced longtime President Ali Abdullah Saleh out of office in February.
His successor, Abed Rabbo Mansour Hadi, was later rubber-stamped as president in a nationwide vote.
Hadi has vowed to fight al-Qaeda while restructuring the armed forces, in which Saleh's loyalists and family members still hold key posts.