The Conservative Party is fighting to keep its General Election campaign on track amid calls for Cabinet ministers to quit.

Boris Johnson, who visited Buckingham Palace for an audience with the Queen before announcing the start of the election drive in Downing Street, faces a first official day of campaigning marred by his party having to defend controversial remarks.

Party chairman James Cleverly said Commons Leader Jacob Rees-Mogg had “caused a huge amount of hurt and distress” with his comments on Grenfell Tower.

READ MORE: Sky News interviews empty chair after Conservative no-show 

Mr Rees-Mogg faced widespread criticism, including from Grenfell survivors and Jeremy Corbyn, after he said people are safer if they “just ignore what you’re told and leave”.

Meanwhile, Kay Burley hosted an 'empty chair' interview after claiming James Cleverley refused to come for an interview. Sources close to Mr Cleverly said that he had not been booked to appear on Sky, and was actually taking part in an interview on TalkRadio at the time of the empty-chair broadcast.

Rapper Stormzy called on Mr Rees-Mogg to resign, calling him a “piece of s**t”, after the Tory MP suggested that Grenfell victims should have used “common sense” and ignored fire service guidance not to leave the burning tower block.

Mr Rees-Mogg apologised for his remarks and said he intended to say he would also have listened to the LFB advice.

Mr Cleverly told the BBC no-one could “credibly” know what decisions those “trapped” in the tower faced that evening.

The Tory chairman also said remarks by prospective Gower MP Francesca O’Brien were “unacceptable” but suggested she would not be axed as Tory candidate.

Ms O’Brien posted on social media that people featured in reality TV show Benefits Street needed “putting down”.

Mr Cleverly said: “This is a comment that the candidate themselves recognised was unacceptable.

“People often tweet in haste and regret what they have done afterwards.”

Mr Cleverly also faced questions about a Tory campaign video that critics claim had been “doctored”.

READ MORE: Conservatives defend 'doctored' Keir Starmer campaign video 

The film depicts Labour’s Brexit spokesman Sir Keir Starmer struggling to answer a question on TV about his party’s position on leaving the European Union, despite him answering it “perfectly adequately”, according to the BBC’s Nick Robinson.

Mr Cleverly told the BBC that Tory social media feeds had posted the interview in full and then followed it up with a “light-hearted” video on Labour’s Brexit position.

“What we also did, and this is not unique to us, is we did a light-hearted satirical video, obviously so with a comedy soundtrack, highlighting the Labour Party’s chaotic position on Brexit,” he said.

Asked whether his party had “posted a lie” online, he replied: “I disagree with your assessment of it.”

Elsewhere, the shadow secretary of state for Wales, Christina Rees, accused Welsh Secretary Alun Cairns of “brazenly lying” about his knowledge of an allegation that a Conservative candidate had sabotaged a rape trial, and called on him to resign.

Mr Cairns claims he had been unaware of former staff member Ross England’s role in the collapsed trial until after the story broke last week.

BBC Wales said it had obtained a leaked email sent to Mr Cairns which showed he had been made aware of the allegations as early as August last year.

The challenges faced by the Tories come on a day when Mr Johnson said Mr Corbyn had taken a stance that demonises billionaires with a “relish and a vindictiveness” not seen since Stalin’s attitude to landowners following the Russian revolution.

General Election 2019Mr Johnson leaves Buckingham Palace (Dominic Lipinski/PA)

Mr Johnson is expected to put Britain’s withdrawal from the EU, the NHS and law and order at the centre of the Tories’ campaign.

He left Buckingham Palace at 10.37am after spending just over 25 minutes with the Queen at her official London residence.

In Downing Street, he is expected to say: “There is only one way to get Brexit done, and I am afraid the answer is to ask the people to change this blockading Parliament.

“It’s time to change the dismal pattern of the last three years and to get out of our rut.

“Let’s go with this Conservative government, get Brexit done, and unleash the potential of our great country – delivering on the public’s priorities of our NHS, crime and the cost of living.

“Meanwhile the alternative is clear – Jeremy Corbyn and his two favourite advisers, dither and delay, turning 2020 into the year of two miserable referendums – one on the EU and another on Scotland.

“And remember that a vote for any other minor party is effectively a vote for Corbyn and his catastrophic political and economic programme.”

Also on Wednesday morning, Ed Vaizey became the latest moderate Tory to announce he would not stand in the election, telling the PM he intended to specialise in the arts, creative industries and technology.