One of the men, Colonel Ian Brazier, interrupted Mr Hammond's speech to criticise him for cutting the Royal Fusiliers Regiment, leading to him being ushered out of the hall by security guards.
His ejection from conference took place as thousands protested outside the conference venue against the Tory-led Coalition's handling of the NHS and a controversial badger cull in England.
Colonel Brazier challenged Mr Hammond to "tell the truth about the disbandment of the Fusiliers".
He added: "I write you letters, you don't respond. The public must know the truth - the Fusiliers are loyal soldiers, you have betrayed them.
"Sir, you need to be looking at defence. This is denial, not defence. You're a disgrace!"
Mr Hammond attempted to persuade the two men to sit down and let him finish his speech, but they persisted in interrupting until they were led away.
Mr Hammond has faced intense criticism for his reforms to the forces, particularly the Army.
Ministers want to cut the number of regulars by 20,000 to 82,000 and double the number of reserves.
The move has proved highly controversial, while recent figures suggest that the Army was struggling to recruit more reserves.
Speaking outside the main conference hall, Col Brazier, chairman of the Fusiliers Association, said he had been speaking on "behalf of all retired fusiliers".
He was accompanied by Captain Joe Eastwood, 76, from Cambridge, when he interrupted Mr Hammond.
Col Brazier, 59, added: "I am very angry about the fact the 2nd Battalion, the Royal Regiment of Fusiliers (2RRF), is to be disbanded.
"We were not in the disbandment plan. We were substituted at the last minute, for reasons we have yet to be told. Freedom of Information requests have been denied, written requests have been denied because it is embarrassing.
"Why? Because 2RRF was the best-recruited infantry battalion in the army when they were selected to be split up - and penny-packeted into less well-manned regiments."
He said Mr Hammond should now be replaced as Defence Secretary by somebody "who knows what they are doing".
Last December, Defence Minister Andrew Robathan apparently tried to get former members of the regiment kicked out of a Commons public gallery during a three-hour debate, claiming they were making too much noise, which he later denied, saying he had the "greatest respect for ex-service personnel".
Mr Hammond last night said the decision to disband the regiment had been made by the Army.
"If they'd stayed in the hall a bit longer they would have heard the case that I was building in my speech for why we have to invest in the new capabilities we need to defend Britain," he said.
Col Brazier hit back, saying: "There are thousands of Conservatives who agree with exactly what we are doing and I'm sorry if Mr Hammond finds that embarrassing.
"It's not half as embarrassing as it is for me looking in the faces of some of our young soldiers who have come back from Afghanistan to be told they are redundant or are going to be posted to regiments they've never heard of."
Murdered Fusilier Lee Rigby belonged to the regiment and Col Brazier said it was "incredibly insensitive" of the Government to disband it.