The Islamists launched an attack on the town of Bama, 45 miles from the Borno state capital of Maiduguri, and were initially repelled but returned in greater numbers overnight.
Sources said there were heavy casualties on both sides and one security source said that as many as 5,000 people had fled.
Two months after Islamist militants in Iraq and Syria declared the area they seized an Islamic caliphate, Boko Haram has also for the first time explicitly laid claim to territory it says it controls in parts of northeast Nigeria.
It captured the remote hilly farming town of Gwoza, along the Cameroon border, during fighting last month. The group's leader Abubakar Shekau in a video declared it a "Muslim territory" that would be ruled by strict Islamic law.
Shekau's forces have killed thousands since launching an uprising in 2009 to carve an Islamic state out of religiously mixed Nigeria, and are seen as the biggest security threat to the continent's leading energy producer and the authority of its president Goodluck Jonathan.
Bukar Auwalu, a trader who fled with his wife, three children and brother, said: "There was firing from different directions."