Pakistani officials identified one of the men as Mullah Obaidullah Akhund, considered the third in command of the Taliban.
The arrest reportedly came only hours after Vice President Cheney left Pakistan, after confronting President Pervez Musharraf with "compelling evidence" of the resurgence of the Taliban and al Qaeda in Pakistan.
"His arrest would be huge, if confirmed," said Barnett Rubin, a New York University professor who testified today before the Senate Foreign Relations Committee on the resurgence of the Taliban in Pakistan.
It was the first time Pakistan had arrested a senior leader of the Islamist militia since it was driven from power in Afghanistan in 2001, and thousands of its fighters fled into Pakistan. As ABC News has reported, the failure of Musharraf government to actively pursue Taliban leaders has long been a sore subject with U.S. officials.
Intelligence sources said Akhund may know the secret whereabouts of Taliban leader Mullah Omar, and perhaps of Osama bin Laden and Ayman al Zawahri.
In the 1990s, Akhund served as the military liaison between the Taliban and the Pakistan's security service, the ISI, which has been accused of sheltering Taliban leaders since the U.S. invasion.
Habibullah Khan contributed to this report.