In his heartfelt paper, The Teaching Of Scottish History In The Scottish Universities, published in 1973, Bruce Lenman commented that, if it featured at all in Scotland's school curriculum, the country's history was generally taught in a "deadly fashion".

My recollection is not so much of its tedium as its invisibility, as if it were a topic that dared not speak its name, or would only come out under cover of dark. I won't be alone in remembering scarcely a lesson that featured Scotland's past. What little I did learn was in primary school, where we were introduced to brochs, the Romans and, most vividly, Mary, Queen of Scots, who had enjoyed one of the most turbulent nights of her life in Dunbar Castle, half a mile from the classroom where I sat.

The very mention of Mary's name is like a red rag to the editors of this magnificent collection of essays, covering the period from 1500 to the present. Tom Devine and Jenny Wormald complain bitterly about the "obsession" among historians with this tragic queen and her "equally lamentable" partner in miserablist drama, Bonnie Prince Charlie. To the fury of serious scholars, both of them have hogged the limelight like karaoke singers who refuse to leave the stage. Yet one can see why. Their tales are the stuff of legend, testimony to the power of personality in shaping the popular view of events. Particularly in the current era where celebrity trounces subtlety, their fascination is enduring.

But those who think there's no more to our past than the exploits of such players, or who were put off Scottish history at school, could do no better than to spend a few hours in the company of the 40 or so historians featured here. Rarely have I read more invigorating, illuminating and inspiring history. By its end, I had to restrain myself from abandoning the office and heading for the national archives to start some of the spadework so badly needed to take the subject further.

When one learns something of the history of Scottish history, as described in Devine and Wormald's feisty introduction, the fact it stands as a subject in its own right at all is surprising. As they write, "well into the 20th century, the intellectual orthodoxies remained unfavourable both to the expansion of research and teaching in Scottish history. Since history had come to be regarded as an important and relevant training for future statesmen, civil servants and imperial administrators, English constitutional history was preferred since it outlined the history of the state in which students were citizens of the United Kingdom rather than Scotland." What research there was tended to focus on the period of Scotland's independence. Astonishingly, as they inform us, "When JD Hargreaves delivered his inaugural lecture as the new Burnett-Fletcher professor at Aberdeen in 1964, he was able to claim that the history of Scotland since 1707 was less studied than that of Yorkshire."

Thanks to a flood of pioneering work in subsequent decades, it is in rude health today – comparatively, at least. The publication of this handbook, and the calibre of historians it includes, testify to that. Yet what may surprise readers of this rigorous volume, and certainly caught me off guard, is that, whether they've read a little or a lot of Scottish history, much of what they think they know is possibly wrong, and very probably only partial.

The great malaise that ails popular Scottish history is a fondness for sweeping, overheated statements. A simplistic view lends itself to soundbites, and plays to our love of catchy headlines and couthy tales rather than awkward and messy fact. These essays, addressing as they do almost every facet of Scotland's past – from the environment and urbanisation to religion, war, culture, diaspora, immigration and nationhood – are anything but reductionist. While crisply and often entertainingly written, offering a concise overview of their subject and the work still needed to be done on it, they manage to convey the depth of complexity in our past that is as yet still only half-understood.

As such, then, this book could not be more timely. As we descend into paroxysms of national self-examination ahead of a vote on independence, it is crucial that we understand how Scotland has become what it is, and why. In the words of David McCrone in his essay here, "Scotland is defined as much by what it has ceased to be as by what it has become." Not since the debate around the proposed Union of 1707 have we needed to be so well informed about the past if we are to make an intelligent choice for the future.

Whether it's a peculiarly Scottish outlook, or common to many small countries, we have an ingrained sense of self-pity. The precise reasons for this attitude remain unclear. That it has worsened in recent times, however, is indubitable. As Colin Kidd and James Coleman write in their breezy overview of Mythical Scotland, "secularisation and the rise of nationalism contributed to the emergence of a mythology of victimhood, grievance and glorious failure running from Flodden (1513), via Culloden (1746) and the highland clearances, to industrial decline. Post-imperial Scotland had become an underdog nation." Tellingly, they cite the fact that "whereas in the 21st century the Culloden battlefield of 1746 signifies a national tragedy, a century and a half ago the site had fallen into serious neglect, with several attempts to raise a commemorative monument ending in failure through lack of interest."

That lack of interest may have been in part due to ignorance of the battle's significance, or disdain for the lost cause it represented, but today's wallowing in the Stuart prince's debacle, and its rewriting in some minds as a simple ideological clash between highlands and lowlands or, worse, between Scotland and England, is unhelpful and even dangerous. One thing we don't want to encourage is the idea that the independence debate is a rearguard action against our nearest neighbour. If it achieved nothing else, this book would correct that view.

Lest all this sounds as though I'm recommending we swallow The Oxford Handbook as a self-improving political pill, rest assured. It's more like a banquet, in which every dish is tempting. In the opening chapter TC Smout, the Historiographer Royal, lambasts the 19th-century – and modern – Scottish fishing industry for its venality: "Despite dissent from some, the dominant voice of science was reassuring," he writes of the herring industry in the 1880s, "and this combined with the boundless enterprise of fishermen to prepare the way for emptying the seas of palatable fish." Of the cod-fishing restrictions imposed in the past few decades, he adds, "Scotland reckoned it got a raw deal in the allocation of catch quotas compared to other fishing nations like Spain, but in the widespread evasion of quotas and landings of 'black fish' the Scots were excelled by none."

In a chapter on The Early Modern Family, Elizabeth Ewan, from the University of Guelph in Canada, throws up some choice illustrations of attitudes towards marriage: "In 1527 Alexander Black sold his right to choose a marriage partner to David Wedderburn in return for £20 and said he would accept any woman David chose, so long as she was free from disease, especially syphilis, and of honourable character, no matter how blind or lame she was." One isn't sure whether to applaud Black's open-mindedness, or condemn his lack of discrimination. Was mutual attraction an irrelevance for our forebears? It seems it might have been, given the number of prosecutions in the 18th century for bestiality. These far outranked rape as a crime, although Anne-Marie Kilday, of Oxford Brookes University, is quick to point out that under-reporting probably explains that statistical oddity.

After the Reformation, the Kirk liked to claim Scotland was more God-fearing than England. A welcome note of scepticism is introduced by Wormald in her spirited chapter, Reformed And Godly Scotland?. Of this often joyless country, in which the Kirk played a pivotal and – it liked to think – iron-fisted part, she comments, "It may be time to move away from concentration on godly success, and think rather about whether HL Mencken's famous definition of Puritanism, 'the haunting fear that someone, somewhere may be happy', might have some relevance to the imposition of godliness in early-modern Scotland - One might go further and ask whether, apart from the mid 17th century, Scotland as a whole was really godly at all." To judge from Kirk session records, where the parish folk seemed hell-bent on continuing to enjoy themselves, not only with their favourite pastime of fornication, but with playing pipes, singing and plays, Scots then were much as they are now – fun-loving and free-thinking.

The breadth of these essays might lead the unwary to assume that Scottish history is a massive industry, pounding away night and day with an army of thousands at its beck and call. In fact, for all its popularity and intellectual standing, it is still a remarkably small academic field. This explains why each of the chapters has suggestions and pleas for areas in which further research is needed, and urgently. As Kilday reflects, for instance, in The Barbarous North?, "it is remarkable that as yet there is no scholarship of male criminality in a country traditionally associated with aggression, confrontation, and displays of masculine authority." More, in fact, has been done on women and crime, and hair-raising it is too.

Across the board, much more study is required, be it in Scottish childhood, or the ecological impact of farming or cities. The challenge for historians is to further the understanding of the country and its place in an international context, without becoming tunnel-visioned and narrow-minded. The great balancing act in historical research is to find the needle in the haystack, and then draw a picture of the society it comes from that's as rich and revealing as a Constable landscape.

Thanks to a handful of groundbreaking scholars, of whom Devine and Wormald are among the most prominent, Scotland has been remarkably lucky in the quality of historians it has attracted and nurtured in recent decades. I would defy the next generation of historians not to find ideas for a career's worth of enquiry among these essays. If Scottish history does not continue to flourish in the next few years, it will not be for lack of inspiration. The same could also be said of the Scottish people.

A word on the price. Many people will want to read this book, and every history teacher ought to, but only some will want to spend £95, even for 600-plus pages. This is a book every library and school should purchase so it is available for everyone.

The Oxford Handbook Of Modern Scottish History

Edited by TM Devine and Jenny Wormald

Oxford University Press, £95