Local authorities have ordered the closure of the Iware market in Ardo-Kola area of the state following the incident on Tuesday.
Taraba police spokesman David Misal said an investigation has been launched to arrest the perpetrators and confirm the motive of the gruesome attack.
So far, no arrest has been made in connection with the killing.
Misal said the information available to the police indicated that the attackers struck in retaliation for an alleged rustling of their cattle.
"The attackers claimed that the cows they (the traders) brought to the market were their cows, stolen during a recent conflict in another area of the state," he said.
Local residents told Xinhua there was pandemonium at the local market as the attackers went on the rampage. Most traders abandoned their wares as they ran to safety.
"We gathered that the killers trailed their victims from Lau, one of the areas where there was a recent farmer-herders conflict," said Jipsari Mohammed, a survivor.
Clashes between herders and farmers in Nigeria have left hundreds dead in the past few months.
The majority of farmer-herder clashes have occurred between Muslim Fulani herdsmen and Christian peasants, exacerbating ethno-religious hostilities.