Ahmad, head of Madagascar’s football federation, won the election in the Ethiopian capital by 34 votes to Hayatou’s 20, official results showed.
Delegates cheered and pumped their fists in the plenary hall after the announcement of the result which heralded the departure of the last of the old generation of leaders in world football.
Ahmad, a 57-year-old father of two, had a discreet playing and coaching career before he took the reins of the Madagascar football federation in 2003.
His rare and determined bid for “change” at the head of the CAF this year took many by surprise, and the incumbent was seen as the favourite.
“When you try to do something, you mean that you can do it,” Ahmad told reporters after the vote. “If I can’t do it, I never stand.”
He earlier said his programme was the “reform of the administration of CAF to avoid the involvement of politics in the organisation.”
Cameroonian Hayatou headed CAF since 1988 and is a senior vice president of FIFA. He was credited with increasing the number of African teams at the World Cup and bringing in extra finance for the continent’s competitions.
“His excellency Issa Hayatou has done a lot for African football (but) it is time for him to step back,” said George Afriyie, vice president of the Ghana Football Association.
Hayatou was led from the auditorium by aides who brushed off requests for comment from journalists while Ahmad was picked up and carried by cheering supporters.
“Africa has made an emphatic decision that we are ready for change,” said Liberian Football Association President Musa Bility.
Seyi Akinwunmi, vice president of the Nigeria Football Federation, called Hayatou a “colossus who has bestrode African football for decades” before adding: “It’s a new dawn, all should be happy.
“We’re Africans and Africans never forget their elders. An elder has been defeated today, but he’s still an elder.
“The world has moved on, we really need energy. And the truth is Issa Hayatou is almost 70. There’s only so much really that he can do.
“The world, technologically and in terms of the way you see things, has changed. Speaking with Ahmad, I found that he understood those things.”