In an appearance before the Parliamentary Commission on Banking Standards, Sir James apologised for his role in the collapse of the Edinburgh-based bank.
He said: "I was horrified and deeply upset by what happened."
Sir James said the financial crisis "should not be the sole explanation" for the problems that led to the bank's rescue takeover by Lloyds TSB in 2008.
He blamed poor loan decisions, particularly at the Bank of Scotland's corporate lending division.
Under questioning from Commission chairman the MP Andrew Tyrie, Sir James admitted that business lending by Bank of Scotland was "incompetent".
He said this "brought down" HBOS.
The combined Lloyds Banking Group required a £20 billion taxpayer bail-out to cope with bad debts.
After the creation of HBOS in 2001 Sir James said the Bank of Scotland corporate arm continued to operate as it had before the merger with Halifax.
Asked if he had had too much confidence in the team that ran the business, Sir James said: "To some extent that must be true.
"They were a highly regarded and experienced team with a terrific track record."
He added: "The risks the corporate bank were taking were not as well understood as everyone thought they were."
Andy Hornby, Sir James's successor from 2006, told the inquiry that "with the benefit of hindsight" HBOS's corporate lending was too concentrated in commercial property and with a small number of large borrowers.
So far former corporate lending chief Peter Cummings has been the only senior banker disciplined by the Financial Services Authority following the crisis. He was fined £500,000 and banned from the industry.
Sir James became chief executive of the Halifax in 1994 and ran HBOS until the summer of 2006.
He said that, unlike former Royal Bank of Scotland chief executive Fred Goodwin, he hadn't given up any of his pension.
He said he had been hit by HBOS's collapse although he had sold two-thirds of his shares between his retirement and the bank's failure.
Sir James said that "it is for others to decide" whether he should be stripped of his knighthood in the manner of the former Sir Fred Goodwin.
He added: "I am in no doubt that my reputation and my achievements will never be seen again in the same light."
Asked if he thought he would be considered a "fit and proper person" by the FSA and qualified to hold a senior job in financial services industry, Sir James said: "I think I am too closely associated with the problems of HBOS for that to be the case."
Sir James insisted he had not sacked Paul Moore, the whistlebower who warned about risks being taken by the bank, saying he had been made redundant.
Ray Perman, author of How HBOS Wrecked the Best Bank in Britain, said: "Clearly the Parliamentary Commission consider Crosby to shoulder a large part of the blame for what eventually happened to HBOS.
"It calls into question in my view whether the FSA can get away with merely putting the blame on Peter Cummings without examining the role of other members of the executive and other members of the board, in particular of James Crosby."