This year's NHS Scotland staff survey found 31% agreed staffing levels were high enough - although this was up 3% from the last time the survey was done three years ago.
Just over half (55%) of the workers questioned agreed that care of patients and other service users was the top priority of the health board they worked for - down 8% from the 2010 survey.
But only 50% of NHS staff would recommend their health board to others as a good place work, a drop of 8% from 2010.
Over the three-year period there was a drop of 11% in the number of staff who said they were satisfied with the sense of achievement they get from their job, with this falling to 60% in this year's study.
Only 26% said staff were always consulted about changes at work, while just under two fifths (39%) said when changes are made they were clear about how these would work in practice.
The survey also revealed almost one in 10 (9%) had suffered physical violence from patients, service users or other members of the public in the past 12 months, while 32% had experienced emotional or verbal abuse.
Over the last year, 11% reported being bullied or harassed by their manager, while 15% had suffered this from another colleague.
Public Health Minister Michael Matheson said: "The staff survey results have given us important feedback on areas of progress, but more importantly where we need to focus our attention to make necessary improvements."
The survey's findings were released at the same time as a new action plan for the NHS was launched, which aims to ensure staff are treated fairly and supported to do their jobs.
The 2020 Workforce Vision plan is based on the views of 10,000 NHS staff, and includes actions on key points raised in the staff survey.
Mr Matheson said: "There can be no doubt that our staff are the heart of our NHS. The many achievements of our NHS are down to them and the dedication they show to patient care.
"As a result we want every member of staff to know that we will do everything that we can to ensure they feel supported to do their job."
A total of 44,389 NHS Scotland employees completed the 2013 questionnaire, some 28% of the overall workforce.
Where comparisons could be made with the 2010 survey, for all but two questions, respondents answered less positively in 2013, suggesting there had been "an overall dip in employee satisfaction from 2010 to 2013".
But for the first time, the majority of NHS staff (52%) said they felt safe to speak up and challenge the way things are done if they had concerns about quality, negligence or wrongdoing.
The survey also found a "high level of employee commitment to their job", with almost nine out of 10 (87%) saying they were happy to go the "extra mile" at work when required - although this was down by 1% from 2010.
Mr Matheson said: "I welcome this feedback which reflects what staff have told us as part of our work to develop our workforce vision.
"This vision is built on what staff feel is not working as well as it should, and where there is room for improvement.
"That is why we have launched our action plan, which takes account of feedback from over 10,000 NHS staff and others, ensuring we are taking direct action as a result of what staff are telling us."
He stressed: "Staff can be confident that we are listening to them and taking all the necessary steps to ensure we have systems in place to support them in their roles and duty to patients.
"Of course there are issues in the staff survey where staff have not felt as valued as we want - particularly around being informed and involved in board decisions."
He said the Scottish Government was "committed to working in partnership to make improvements", with Health Secretary Alex Neil having written to health board chairs setting out what action the Scottish Government will take at a national level and what ministers expect boards to do.
Mr Matheson said: "The Cabinet secretary has made clear that boards have a duty to listen to what staff are telling us, and to take action to address the issues which are of concern to them."