The MP for Glasgow South was appointed shadow environment, food and rural affairs minister in May last year.
But he told party leader Ed Miliband that combining that extra role with his responsibilities as a husband and father had proved "beyond me".
In his resignation letter, the former minister who sought the leadership of the party in Scotland, said he had been "delighted" to return to the frontbench.
But he added: "You and the party need frontbenchers who can fully commit to holding the coalition to account and to doing the hard work necessary to move into government in two years.
"I am faced with the uncomfortable truth that my talents, such as they are, do not extend to being an effective front bencher as well as a good husband and father."
Mr Harris, an MP since 2001, has two sons with wife Carolyn as well as an older son from his first marriage.
He said he was not advocating making the Westminster role less time-consuming, saying it was right that being an MP should be a "tough job".
"Most shadow ministers with young families seem to cope quite well in balancing the demands of their jobs with their family commitments. I envy them, because I've tried and I can't," he wrote.
"So I'm passing on my briefings and papers, my diary appointments and contacts, my facts and figures about wildlife cruelty, fishing quotas and bee action plans, and my badger costume (that last one was made up but it's an idea) to my lucky successor.
"I genuinely hope he or she becomes a Defra minister. As for me, I'm looking forward to spending more time with my family. Serves them right, too."
Mr Miliband MP said: "Tom is a huge asset to Parliament, the Labour Party and the people of Glasgow South.
"I totally respect his decision to step down from the frontbench, on which he served with genuine distinction. It is the right choice for him and his family.
"He will continue to serve the Labour Party and his constituents from the backbenches with his unique passion and wit.
"And I look forward to working with him as we campaign for a Labour government at the next general election."