Paul Kingsnorth's The Wake, written in an invented language based on Old English, was published by crowd-funding publisher Unbound.
Smith, originally from Inverness and now based in Cambridge, has been short listed for How To Be Both, the third time she has made it onto the £50,000 prize's short list.
There are only three female writers on the 13- strong list: Smith, Karen Joy Fowler, for We Are All Completely Beside Ourselves, and Siri Hustvedt, for The Blazing World.
David Mitchell, David Nicholls and past winner Howard Jacobson are also in the running for this year's prize.
A rule change has now opened the award to more international writers, enabling Americans to be included for the first time, with four US authors on the list.
There is also room for Australian Richard Flanagan and two Irish writers Joseph O'Neill and Niall Williams.
They would have been eligible under the previous rules which allowed authors from the Commonwealth and the Republic of Ireland.
Organisers of the award announced last year they were opening up the 46-year-old prize to writers of any nationality writing in English.
Jacobson, who won the Booker in 2010 for The Finkler Question, has made the 2014 longlist for his book J.
The chair of the judges, philosopher AC Grayling, said: "This is a diverse list of ambition, experiment, humour and artistry.
"The novels selected are full of wonderful stories and fascinating characters."
Last year's prize was won by New Zealander Eleanor Catton, the youngest winner of the prize at the age of 28. Her book,
The Luminaries, at 832 pages, was also the longest winner.