Sitting MSPs will face the axe if they fail to win the support of more than 50% of members in their area.
A senior party source said the new rule effectively meant under-performing List MSPs were "vulnerable" to being ditched.
The 2011 election saw a host of well-known Labour MSPs lose their seats and a raft of unfamiliar List members returned. Although a number have impressed, others have struggled to make an impact and, as a result, Labour considered reform of reselection procedures that would make it harder for incumbents to stand.
List MSPs are "protected", which means once they are reselected no-one else in the party can get a higher place in the rankings than them.
Proposals to change this system were abandoned following resistance by sitting MSPs. However, the Sunday Herald has learned that the party approved a separate reform last month that places an obstacle in the way of MSPs standing again.
In the past, Labour MSPs sailed through a re-selection process that gave affiliated trades unions more, disproportionate, power than ordinary members.
An electoral college has been agreed for reselections: 50% of the vote goes to members, 50% to affiliates.
To win the whole member section, sitting MSPs must secure support from at least half of the members in their constituency or region. List MSPs who fail to clear this hurdle cannot get a "protected" place and are effectively demoted.
Several List MSPs are expected to find it hard to win this level of support. Although the system is the same for constituency and regional members, first-past-the-post MSPs are not expected to slip up.
Sources said the new college empowered members at the expense of affiliates, due to the union section not being a block vote.
Its 50% section is divided into the number of affiliates registered in the region or local constituency.
One insider said: "Well-respected MSPs won't be worried about these changes, but those who have not delivered at Holyrood will be concerned. They are vulnerable."
Another source said: "I think some List MSPs are going to struggle, and rightly so."
Following the 2011 electoral rout, a review gave the Scottish party full control of the selection process for Holyrood elections.
A Scottish Labour spokesperson said: "Our party members live, work and campaign in our communities and they know best who should be representing them. This is the latest in a series of measures to empower our loyal and hardworking members."