Official figures show the percentage of pupils in the fourth year of primary who were performing either "well" or "very well" dropped from 76% in 2011 to 69% last year.
The National Numeracy Survey, of 10,561 pupils, also showed the proportion of P7 pupils doing "well" or "very well" fell from 72% in 2011 to 66% in 2013.
Although performance did not get worse over the same period in secondary, the proportion of pupils performing at higher levels was just 42% in S2 in 2013.
The attainment gap between children from the most and least deprived areas also grew wider.
The decline in numeracy is particularly worrying because it comes at a time when Scotland has introduced reforms to what is taught under the new Curriculum for Excellence (CfE).
The Government said it would make an extra £1 million available to fund school numeracy hubs to help drive up standards.
However, opposition politicians and teaching unions seized on the findings, arguing the numeracy hubs would not be sufficient.
Liam McArthur, education spokesman for the Scottish Liberal Democrats, blamed larger class sizes in primary and cuts to the number of teachers for the decline.
He said: "Parents will be concerned that the figures are going in the wrong direction on Education Secretary Michael Russell's watch. This is a basic skill which our children need to get on in life, and it is alarming that in the space of two years the percentage of pupils performing well or very well has dropped."
Mary Scanlon, education spokeswoman for the Scottish Conservatives, said the statistics were proof "all is not well" in the education system, while Kezia Dugdale, Scottish Labour's education spokeswoman, added: "These figures are a wake-up call for the Scottish Government. We can't be complacent and we can't let this deterioration continue."
Larry Flanagan, general secretary of the Educational Institute of Scotland (EIS) trade union, said the figures highlighted the need for change.
He said: "Although the drop is relatively small and there are a number of potential mitigating factors, it does serve to underline the point that there remains an ongoing need for additional support to embed CfE in the primary sector.
"The current focus on the new qualifications has often led to the mistaken believe that the process is complete in primaries, when clearly it is not. One area which clearly needs to be looked at is the transition from primary to secondary, and that needs both sectors to work together."
Alan McKenzie, acting general secretary of the Scottish Secondary Teachers Association, said the situation was also worrying in senior schools.
He said: "Many of our members complain of poor attainment levels in numeracy being demonstrated in S1.
"It is always difficult to capture this kind of perception, but I think we have something here of which we need to take full recognition.
"In addition, the data around the influence of poverty and deprivation on numeracy attainment will be alarming for the Scottish Government."
Dr Alasdair Allan, Minister for Learning, said the issue would be tackled by an expansion of the numeracy hubs programme.
The Government is planning to expand the existing six hubs and create additional centres under a £1.2 million programme.
He said: "The picture that emerges in these figures is one of very few pupils at P4 or P7 not working within the expected level, yet there is a clear issue by S2.
"Pupils who come from the poorest households are lagging well behind those who come from wealthier backgrounds."
Meanwhile, the Royal Society of Edinburgh has launched a web resource for young people aimed at encouraging them to be confident with numbers.
It features videos of members of its Young Academy of Scotland showing how they use a variety of number skills on a daily basis.