A senator in Borno state said troops had abandoned the base in the town of Baga after it was attacked on Saturday.
Residents of Baga, who fled by boat to neighbouring Chad, said many people had been killed and the town set ablaze.
Baga, scene of a Nigerian army massacre in 2013, was the last town in the Borno North area under government control.
It hosted the base of the Multi-National Joint Task Force (MNJTF), made up of troops from Nigeria, Chad and Niger.
Set up in 1998 to fight trans-border crime in the Lake Chad region, the force more recently took on Boko Haram.
Boko Haram attacks towns and villages on an almost daily basis, abducting people including young boys and girls, BBC Africa analyst Mary Harper reports.
The military, which includes Western advisers and surveillance, seems incapable of dealing with the problem, she adds.
Residents who fled to Chad said they had woken to heavy gunfire as militants stormed Baga early on Saturday, attacking from all directions.
They said they had decided to flee when they saw the multi-national troops running away.
Maina Maaji Lawan, senator for Borno North, told BBC World Service civilians had run "helter skelter" - "some into the forest, some into the desert".
Communications with the town were cut off and exact information about casualty numbers could not be confirmed, he said.
"We are very dispirited," the senator added.
Confirming that the military had abandoned the base, he said people's frustration knew "no bounds" over the apparent fact that the military had not fought back.
"There is definitely something wrong that makes our military abandon their posts each time there is an attack from Boko Haram," the senator said.
Related stories: Boko Haram kidnap 40 boys and young men in remote village