It's a brave director who takes on a prison movie.

On the one hand, it's a genre that can be as limited and boring as life within prison walls; on the other, it has produced the film that tops a younger generation's favourite film list, The Shawshank Redemption - and you're on a hiding to nothing to try to beat that. But Scotland's David Mackenzie is a doughty fellow. And with Starred Up he's got a couple of aces up his sleeve - a script by someone who has worked inside the prison system, and a pair of cracking performances.

Eric Love (Jack O'Connell) is a 19-year-old with an attitude to match his bleak record. As we watch him go through the usual drill of entering prison - the strip and body search, the chilly welcome by the guards and a cell that you couldn't swing a cat in - his swagger suggests he's accustomed to such places. The fact is, this is Eric's first day with the grown-ups.

The young man with the ironic surname is being "starred up", brought forward from youth detention to prison ahead of time, because he's trouble. And he has no intention of letting up. His first act is to make a knife out of the ­innocuous items in his wash kit. It isn't long before he's assaulted another prisoner and invited a gaggle of guards for some face time.

And yet Eric has people watching out for him. One is fellow tough-nut prisoner - and estranged father - Neville (Ben Mendelsohn), who is keen to make up for his errant parenthood by keeping his son out of trouble. The other is prison psychotherapist Oliver (Rupert Friend), a sensitive soul whose anger management classes are proving a hit with some of the inmates.

Boy, do they need it. As usual with the milieu, the testosterone in Starred Up can be wearing and alienating; it's difficult to sympathise with characters who won't stop swearing and strutting.

Interestingly, scriptwriter Jonathan Asser, who worked himself as a prison therapist, turns this problem into a virtue, by making anger management - in a place where it really doesn't pay to annoy your fellows - one of his principal themes. The character dynamics are set within this thematic framework - between father and son (who will need to take a few chunks out of each other), inside Oliver's class, between inmates and guards, and finally between Neville and the prison's controlling inmate, whose velvet-voiced claim to "want a nice, quiet wing" is more menacing than reassuring.

The question is, at what point in the chain will the pressure finally blow? While we're wondering that, Mackenzie keeps things moving at a good pace - his camera eagerly following the bristling Eric as he roams the cell block making mischief - and allows his actors to make their mark. There's a great deal of originality in the father-son relationship, and both O'Connell and Mendelsohn ooze bad-boy charisma. The Australian Mendelsohn plays Neville like an ageing bear, at his most dangerous when the worse for wear; and O'Connell skilfully conveys a glimmer of needy humanity beneath the hard-nut exterior.

The younger actor will soon be seen in another Scottish-made film, Troubles drama '71. But this is the better showcase for his ­potential, and brings to mind another borstal boy breakthrough role, that of Ray Winstone in Scum. Starred Up is not for the squeamish. And it doesn't match the genre-transcending quality of my own favourite prison movie, Jacques Audiard's A Prophet. Nonetheless, it's an admirable piece of work.