As a singer-songwriter working in a genre, Americana, where it is common practice for two or more writers to book meetings where they will write songs together, Baker is a loner.
Words have to fit his singing style - which he concedes is not graced with wonderful technique - or he will not sing them. A phrase he is not happy with will not be sung. Whole songs have been discarded because of a rogue word Baker did not like but somehow could not replace with another word that felt natural.
But if you think the Austin, Texas-based Baker's songwriting is fastidious, you should try getting him to do a painting for you.
"Oh boy, I am worse at completing paintings and drawings than I am songs," he says, having just arrived for his latest UK tour.
"People commission me and they don't take delivery, they get visitation rights. It is like an estranged dad coming to see how his kid is getting on, getting to take the painting out in the car for a run. Well, it might be if I can find it - my place is chaotic, with walkways through canvases. But it is like songwriting, if I could master the 15- minute song, I would have a huge catalogue. If I could paint quicker, I might be able put a plate of food on the kitchen table without having to move painting materials - or song lyrics."
Baker has earned the right not to hurry things. His response to the trauma that affected him when a terrorist's bomb exploded on the train that was about to take him to Machu Picchu in 1986 and left him at the mercy of the surgeon who repaired his femoral artery and saved his life might have been different.
He could have treated his time as something that had to be occupied at every opportunity, tried to achieve as much as possible. That would have been an understandable reaction.
Instead, he has realised him ambition gradually. When he last spoke to The Herald, in 2007, Baker was just beginning to make an impression in America and was about to play overseas for the first time.
Never, as he would freely concede, a virtuoso guitarist, he had managed to switch to playing left-handed due to injuries received in the bomb blast. He made an album, Mercy, in 2004, that he assesses now as a good album in a world full of great records that get attention.
He was still working on a building site at the time and had had to overcome stage fright that was so bad he could hardly speak. Audiences who have caught him subsequently might find this hard to believe - he could talk for Texas on stage now.
After a year and a half of gigs at a roadhouse-diner on the outskirts of Austin, where people were initially more interested in eating and getting drunk, his stagecraft developed to the extent he could get the audience on his side and listening to his songs, which he sings in a very individual, compelling way.
A second album, Pretty World, in 2007, reached a wider audience at home and found favour with influential radio presenters on both sides of the Atlantic, including Radio 2's Bob Harris.
Baker's gig diary started filling up and despite suffering from hearing loss, another effect of the bomb blast, that left him completely deaf in one ear and with only 50% hearing in the other, Baker was able to cope with working on festivals, where soundchecks can be minimal. Eventually, he was even able to afford the luxury of touring with a band.
"My hearing difficulties are tough but everybody has problems, you just have to get on with things," he says. "I have a hearing aid that lets me hear all the high end stuff and closes out all the outside noise and I have learned to position myself so I get the best out of the on-stage monitors.
"Plus, as long as I can feel the kick drum, I can stay on the beat. It has really been as much - or more - about the band accommodating me than me accommodating them, and other than singing and providing a foundation, I stay out of the way and let them do all the beautiful things."
For the tour that brings him back to Scotland this weekend, Baker is reverting to just voice and guitar, a modus operandi that allows him the freedom to re-introduce songs that have dropped out of the band set-list. He can also go off-script if the urge strikes, although he has come through the phase that saw him flirting with stand-up comedy a visit or two back.
"I still chat to the audience, of course," he says. "But there was a danger for a while that the humour might detract from the songs rather than enhance them. I have a new electric guitar, which I can hear better, so I am really enjoying working with that and getting the songs across. But then, I just enjoy my work generally these days because any day that is not spent on a construction site I am in paradise."
Sam Baker plays Blue Lamp, Aberdeen, tomorrow and Pleasance Cabaret Bar, Edinburgh, on Saturday