Alex Garnett isn't the type to join exclusive clubs.

But if he was, he could start a couple of his own. The London-based saxophonist, whose CV includes work for Bryan Ferry and Jools Holland among many other luminaries, could form the League of Jazz Musicians Who Trained as Actuaries with Chris Barber. He's also the second generation of his family to play with the Rolling Stones - his dad, Willie, being a legend among musicians on the London scene both as saxophonist and saxophone repairer - and not many families can make that claim.

Contrary to a common assumption, it was Garnett's mother, a piano teacher, who set him on the road that would take him to blues singer-guitarist Otis Grand's band and on to jam sessions with blues greats B.B. King, Albert Collins and Buddy Guy as well as sharing stages with Jimmy Page and Duck Dunn, of the Stax Records house band and the Blues Brothers, before going on to make, belatedly, one of the most accomplished debut albums in British jazz.

"My old man stayed in the background and never forced the issue where playing music was concerned when I was growing up," he says. "But I think my mum had high hopes for me as a pianist. There was always a piano in the house, because she taught, and I played around on it a bit. Then, when I got a soprano saxophone at the age of seven or eight, she made sure I went through the grade exams. Which was and is a blessing but I had a rude awakening in my early teens when I realised that I could read any piece of music you put in front of me but I couldn't busk Happy Birthday."

It was at this point that dad stepped in. He advised Alex to listen to jazz and buoyed by the news that jazz musicians made up all this amazing stuff he was hearing on the spot, Alex went off and studied Ben Webster, Lester Young and a bit later, Michael Brecker. He'd never seen music as a career, however, and at 16 he got a job in the Stock Exchange as a trainee actuary. There he might have stayed but for the 1990 stock market crash. Deemed surplus to requirements, he was laid off with a wodge of cash and took himself off to south-east Asia on a one-way ticket.

After a year of wandering around jungles and "seeing how the other two-thirds live", he returned to London and took a job in a music shop. A colleague persuaded him to take up the saxophone again and within a few weeks he had joined Otis Grand, having excelled as a dep on a couple of gigs. Between the contacts he made himself and the musicians he hooked up with through them bringing their instruments to the house for his dad to repair, Garnett soon earned a reputation as the sort of hired hand who can fit into any musical situation.

He laughs as he talks about one job that came about after a painter and decorator his dad had hired turned out to have morphed into Ray Gelato, leader of the 1990s Jump Jive & Swing revival, with whom Garnett toured the world for 12 years. Garnett was happy to be the eternal sideman until the American bass player who had joined Gelato, the tireless musician-impresario Mike Janisch, suggested that a recording Garnett had made at Janisch's behest as an expensive demo in New York should be the first release on Janisch's Whirlwind record label. Serpent, as the album was named, made many critics' best albums of 2011 lists, and despite being reasonably pleased with it, Garnett jokes about still having boxes of it around the house and considers Janisch's suggestion of a follow-up "over-exposure".

"I never really saw myself as a recording artist," he says. "I've played on innumerable sessions and noticed that people often play much better on other musicians' records than they do on their own. So I suppose I was dodging the pressure until Mike twisted my arm. The other thing I've noticed is that the music often sounds much more fully developed at the end of a tour than it does on the CD you're touring to promote, so I'm reversing the process. I've brought in a friend from New York, Tim Armacost, who's a great saxophone player, and formed a quintet this time. We're going to tour the new music first and then record it, and I'm hoping we wind up with something that gives a true representation of what develops on the road."

Alex Garnett's Bunch of Fives play the Jazz Bar, Edinburgh, tonight and Glasgow Art Club tomorrow.