Today, and every Saturday of the year, impeccably dressed women who know how to wear a hat with insouciance will momentarily stop in the south side of Glasgow, open their designer handbags and allow a young man to forage briefly among the contents.

The fact he is wearing a high-vis jacket suggests he is not an aficionado of clasps and leather stitching, but instead a security guard ensuring no bombs or firearms are secreted inside the bag. In modern Scotland it is hard to imagine that, for some people, the simple act of worship still has overtones of danger as security guards are employed to forestall acts of violence at the country's synagogues.

This is the reality of being a Jew in Scotland, of belonging to a people who fled strife around Europe and settled in this country and made an astonishing success of their lives within just a couple of generations, building businesses known throughout the country, entering and contributing to many strata of civic Scotland, yet still feeling in many cases that they were never fully accepted. Like a comet flashing across the sky, they have only been here for a century yet already their numbers are falling. At their peak there were 15,000 Jews in Scotland but there are fewer than 5000 now, and the white hair of the gentlemen sitting in shuls - the word "synagogue" is rarely used - suggests the number will continue to drop.

Jewish traders had been visiting Scotland for hundreds of years, but the real period of immigration was at the beginning of the last century when Russian and Polish peasants fled the pogroms in their countries, escaping from ruthless murders and village burnings that threatened their very existence. Ships from the Baltic sailed to Dundee and Leith, where those with a few more coins took the train to Glasgow to sail onwards to New York and a new life in the new world.

Many Jews like to tell the tale that their families were so unworldly that unscrupulous crew members informed them that Dundee was New York so they would go ashore, and the crew could sell on their berths to Scottish migrants. In truth that probably didn't happen to many, but why waste a good story? Many of the Jewish immigrants with little money arrived in Glasgow, either to live or to save money before continuing their journey to America. They settled in the Gorbals, where accommodation was cheap and plentiful if not exactly sanitary or healthy. Those with skills from the forests of Russia became furniture makers, others tailors and jewellery makers. Some were pedlars, buying goods from warehouses to sell round the doors in Glasgow, Lanarkshire and further afield. The successful ones then opened their own stores.

With shipbuilding and engineering booming in Glasgow, a new middle class was emerging with money to spend and Jewish stores such as Goldbergs and Sellyns introduced buying on credit so that Scots could finally afford fashionable furniture and clothes, paying them up weekly at a time when credit cards did not exist.

As their prosperity grew the Jewish community began to move south from the Gorbals, freeing it up for the next wave of immigrants, this time from Ireland. They settled first in Queen's Park and Pollokshields, then Giffnock and Newton Mearns. Many were perhaps not the most religious of Jews. Not for them the long beards and permanent skullcaps of those whose Jewishness controlled every aspect of their lives. Many wanted to assimilate themselves as much as they could into Scottish lives, so much so that both Rangers and Celtic have had Jewish directors. Nowadays, Jewish boys at their bar mitzvahs will often have special kippas, or skullcaps, in the colours of their chosen Old Firm team.

Even for less religiously observant Scottish Jews, the culture they have inherited remains a key part of their lives. At the centre of it is family and Friday night dinner. Friday is when the family come together in a joyous if noisy gathering of generations to sit down, talk and argue endlessly, and eat. As someone who is not Jewish but who married into a Jewish family and has been invited to many such dinners across the south side of Glasgow, this is how I find many unfold.

They will start with homemade chicken soup (sometimes the cliches are true) and even that brings a gabble of voices, discussing where the hen for the soup was bought. "It seemed awfully small, I don't think the quality is there any more," the matriarch will declare, although that should then be met with a chorus of denials that the soup is anything less than perfect. Then there will be shouts of whether you want it with or without vegetables, with or without knaidel (a type of dumpling), with or without lokshen (strips of pasta) so that even a bowl of soup has endless possibilities and arguments.

Then comes the organ recital - a list of who has been in hospital to have operations on various organs. Most Jews assume a serious illness is only moments away and will discuss who is, or is not, a good surgeon to see, in the way football fanatics discuss the merit of recent signings.

Three conversations will be going around the table at once, but everyone can hear every conversation and will dip in and out of whichever one takes their fancy. If someone is mentioned in a story, the teller will be interrupted while the lineage of the person mentioned is discussed. Something along the lines of: "Ivor Cohen had a miraculous escape from an accident." "Wait, is that Sylvia Cohen's son?" "No, Sylvia's son moved to Manchester." "Oh, whereabouts in Manchester?" The miraculous escape of Ivor Cohen will have to wait another half-hour until the conversation finally returns to the matter.

Then education will be discussed - whose children are doing well in school and who has gone to university. "She got six As." "Well, that's a private education for you." "No, they had to get her tutors. I don't know why they were paying for that school." And so the arguments swirl around.

In fact the emphasis on education is one of the reasons the Jewish community in Glasgow is unravelling. The younger Jews leave for university, often in London, Manchester or Leeds, never to return, as the job prospects are never quite as good in Glasgow. "They've seen the bright lights and Glasgow seems dull by comparison," David Strang tells me.

Strang, a senior member of the Jewish community, was originally named David Sragowitz, but like many changed his surname. Not, he says, because of anti-semitism, but simply because people could not pronounce or spell his name, and it made things simpler. "Nobody could pronounce it," he says. "It would sometimes come out as 'scraggy bitch'." As a pupil at Shawlands Academy in the 1940s, he blended in with the other pupils and his Jewishness was not remarked upon, he says.

If there was discrimination in Glasgow in the 1950s it was at the golf clubs where Jews - and often Catholics - were not welcomed as members. Finally a group of Jewish golfers bought their own land on the moors at Bonnyton and built their own golf club, where, thanks to their socially inclusive nature, the membership is less than one-third Jewish.

Although the Jewish community might seem remote to other Glaswegians, many Jews have enthusiastically given back to the country that let them in. Few bar mitzvahs, weddings or formal dinners begin without a toast to the royal family, listing the main members of the royal household so the toast is not simply a second-long afterthought.

Then there are the charitable contributions. Jews themselves will be the first to joke about Jews being careful with their money, but hospitals, medicine and charities have gained millions of pounds from Jewish foundations in Glasgow such as the Wolfson family who founded Great Universal Stores, or the Walton Foundation, still overseen by the accountant David Walton who, with his father Isidore, agreed to set millions aside for charity from their Scottish Metropolitan property company. In fact, surgeons at Glasgow hospitals were so pleased with the equipment the Walton Foundation bought for them that they insisted David, whether he liked to or not, scrub up and join them in the operating theatre to see how they were using it.

The lives of today's Jews are captured in Judah Passow's book of black-and-white photographs, showing the smiling faces at a wedding, the anguish of children visiting the Anne Frank museum in the Netherlands, a proud bar mitzvah boy waving to his mother in the gallery, the stark reality of a funeral. It is a familiar, yet slightly different world, Scottish but with a flavour of Jewishness. Thus a guest at a wedding wears not only a kilt but also a kippa. The groom is surrounded by his friends, but he is being tossed in the air from a chair held aloft by the male guests.

All this should be a time of happiness in Scotland's Jewish community, with families prosperous and children doing well, but some feel a cloud hangs over them. Israel has a special place in many Jewish people's hearts as their homeland, the one country that will welcome them after so many European countries have rejected them. But conflict in Gaza has heightened anti-semitism across Europe and in Scotland. There are thousands of civilians being killed in Syria, Iraq and Afghanistan without a moment of protest in Scotland, many Scottish Jews argue, but if Israel defends itself then it is publicly condemned.

As the writer Michael Mail points out in the introduction to Passow's book, "There are concerns that some of the more extreme forms of anti-Israel sentiment found today in Scotland reflect darker motives. There have been 39 motions on the subject of Israel in the Scottish Parliament over the last three years, while the war in Afghanistan, in which Scottish soldiers fought and died, generated just three."

The late artist Hans Jackson, who settled in Glasgow, created striking paintings of Nazis smashing the shop windows of Jewish shops in Germany. Now the Jewish Telegraph reports that the Kedem cosmetics stall at the St Enoch centre was forced to close after being besieged by anti-Israel protesters. There were claims that the same happened at the Jericho stall selling products from the Dead Sea at Silverburn in the south side of Glasgow, although the shopping centre's management later said it closed as its lease had run out. As one Jewish woman said to me, "When did the Jews in Germany know it was the right time to leave?"

Many Jews have placed "No thanks" posters in their windows ahead of this week's independence referendum in the belief Scotland is a better place as part of the United Kingdom. It would diminish Scotland as a welcoming country if the Jews now felt "No thanks" applied to them as well.

Scots Jews: Identity, Belonging And The Future by Judah Passow is published by Bloomsbury, priced £25.