Vice President Yemi Osinbajo will instead visit the restive southern region to launch a cleanup program of the Ogoniland, an area badly hit by oil spills, the source said, without giving a reason for Buhari's cancellation.
Buhari had already skipped a visit to the commercial capital Lagos in the south last month at the last minute. Posters with his picture had been already hung up to welcome the president before his spokesman cited "scheduling" difficulties.
It would have been the first visit of the former military ruler to the Delta since taking office in May last year. Critics have accused Buhari, a Muslim from the north, of ignoring the predominately Christian south.
The southern Delta swamps have been hit by a series of militant attacks on oil and gas pipelines which have brought Nigeria's oil output to a 20-year low.
Hours after the announcement of Buhari's visit to the swamps on Tuesday the Niger Delta Avengers militant group, which has claimed several attacks on Chevron and Royal Dutch Shell facilities, issued a warning to oil firms that their "facilities and personnel will bear the brunt of our fury".
The Avengers have accused Buhari of ignoring local problems.
Buhari said on Sunday the government would hold talks with leaders in Nigeria's main oil-producing region to address their grievances, in a bid to stop a surge in pipeline attacks.
Residents in the swamp areas have for years complained about oil industry pollution and about economic marginalization by the government.
Local officials and Western allies such as Britain have told Buhari that moving army reinforcements to the Delta region would not be enough to stop the attacks and that the population's grievances should be dealt with.
Related stories: Nigerian military warns Avengers to end hostilities