It's not often you come across an artwork that makes you shudder.

This happened to me last year when I spent an afternoon on my own in late summer, walking through Jupiter Artland's green and pleasant pathways.

This outdoor gallery on the outskirts of Edinburgh has some impressive works, but the one which hit me between the eyes was Ian Hamilton Finlay's Only Connect. These words – the last in EM Forster's novel Howards End, which explores the theme of how different worlds have to find common ground – are carved into two milestones either side of a small arched bridge made of Northumbrian limestone.

Finlay, who died in 2006, aged 80, was first and foremost a poet and an intellectual. In the 1960s, he became one of the leading lights in the Brazil-based concrete poetry movement, which saw poems become solid pieces of art which you could touch and feel. He also wore many other hats, including that of philosopher, short story writer, artist, gardener, classical scholar and agitator with a wicked sense of humour.

As a new exhibition at Glasgow's Gallery of Modern Art (GoMA) shows in spadefuls, Finlay had a crystal-clear vision of what he wanted to convey in his art, using words as a weapon. On show alongside Glasgow Museums' collection of Finlay prints are two installations specially created for GoMA when it opened in 1996. Both relate to the French Revolution which, along with war and a passion for the values of neo-classicism, are recurring themes in his work.

The most eye-catching is a blackly comic repost to personal slight by three individuals: Sunday Times art critic Waldemar Janusz-czak, gardening writer Gwyn Headley and the influential French intellectual Catherine Millet. The installation consists of three heads, made by sculptor Alexander Stoddart for Finlay, which languish in straw baskets, accompanied by verse by leading French revolutionary Robespierre: "We want to substitute morality for egotism, duty for etiquette, integrity for insolence, large-mindedness for vanity merit intrigue."

The prints and sculpture here also include a recent donation of new prints by Pia Smig, executor of Finlay's estate. Cards, prints and booklets lay at the centre of his work. All his outdoor and indoor installations reference the works on paper.

Finlay was born in Nassau in the Bahamas to Scottish parents. His father smuggled bootleg alcohol from Nassau into the US until the repeal of prohibition laws in 1933 and sent young Ian to boarding school in Scotland. The sense of being an outsider remained with Finlay for life. He attended Glasgow School of Art briefly in the early 1940s before being drafted with the Non-Combatant and Service Corps between 1944 and 1947. A spell in post Nazi-Germany left a lasting impression on him.

According to exhibition curator Sean McGlashan, the prints on show here offer a rare opportunity to see Glasgow Museums' substantial holding of Finlay's work. "These were hidden in plan chests for years," he says. "They have now been reframed and look incredibly fresh. Among work donated by Smig, there is a drawing of a proposal for the new GoMA in 1996, which shows windows with the words, 'liberty', 'equality' and 'fraternity'."

Much of Finlay's work, says McGlashan, is about railing against power. Alongside this constant questioning, is humour – often black – as seen in the lithograph, A Panzer Selection (1975), which shows tanks printed in gold on brown paper to imitate a selection card found in a box of chocolates.

In Neoclassicisme Revolutionnaire, Finlay plays with irony and time by stating the French Revolution began in 1789 and ended in 1989, the year after the print was made. As with much of Finlay's work, there is history here. Following a public outcry by Catherine Millet, the kibosh was put upon a 1987 commission by the French Ministry of Culture to design a garden commemorating the Rights Of Man for the bicentenary celebrations of the French Revolution. Millet was the leading intellectual, you will recall, whose metaphorical head Finlay chopped off and placed in a basket at GoMA -

What I loved about these prints is the simplicity of Finlay's vision wrapped up with a hard edge of intellectual rigour and playfulness. Finlay isn't afraid to wield his scythe to sever the thinnest blade of grass.

Ian Hamilton Finlay: Poet, Artist, Revolutionary, Gallery of Modern Art, Royal Exchange Square, Glasgow (0141 287 3050, www.glasgowlife.org.uk), June 20-March 1