The incident sparked widespread anger in what one Senator described as a "siege" on the country's democracy.
Video of the incident shows Nigeria's lawmakers in angry confrontation with the masked men who blocked all gates leading to the parliament, denying senators and journalists access.
Local media reported that the men were operatives of the State security service headed by Security chief Lawal Musa Daura, who has now been fired.
Several of the lawmakers, mostly from the opposition party, posted images and videos of the blockade, which happened early Tuesday morning.
Senator Ben Murray-Bruce claimed legislators from the opposition were locked out of their offices in the parliament, while those from the ruling party had earlier gained access into the building.
"APC (All Progressives Congress) senators are now in the chambers strategizing. They want a change of power, and they are desperate for leadership. They are now using law enforcement agencies for their bidding, but we are documenting all those involved in this illegal activity, " Murray-Bruce told CNN.
President Buhari is currently out of the country on a 10-day working vacation in the United Kingdom, and his Vice President Yemi Osinbajo is ruling the country in his absence.
The security officials were acting without Osinbajo's authority as his spokesman, Laolu Akande, earlier told CNN he did not know why the lawmakers were prevented from entering the building.
"This administration is fully committed to the principle of separation of powers and will not do anything against those principles or against the constitution," Akande told CNN.
Akande later released a statement saying that Osinbajo described the unauthorized takeover as a "gross violation of constitutional order, rule of law and all accepted notions of law and order."
According to him, the unlawful act was done without the knowledge of the Presidency and was "condemnable and completely unacceptable."
However, opposition party legislators claim the incident was motivated by a desire to impeach the Senate President Bukola Saraki who recently defected from the ruling APC party to join the main opposition, People's Democratic Party (PDP).
Saraki, Nigeria's third-most senior politician, left President Muhammadu Buhari's party after weeks of speculation over his loyalty.
His announcement followed mass defection of top politicians from APC to PDP in recent weeks.
President Buhari is seeking reelection early next year but has been beset by problems with rising insecurity in the country as well as a lack of confidence in his leadership which has led to a large number of defections from his ruling party.