It said the Security Council had failed in its duties towards Syria as well as in other world conflicts.
Saudi Arabia has previously expressed frustration at what it sees as an international failure to act on Syria, where it staunchly backs the rebels.
There has been no official reaction from the UN, but diplomats there expressed surprise at Riyadh's announcement.
Russia's foreign ministry called the move bewildering, and said Saudi Arabia's criticism of the UN Security Council about its actions over Syria "is particularly strange".
The announcement came hours after Saudi Arabia was elected for the first time to one of the 10 rotating seats on the Security Council.
The non-permanent members sit on the council for two years, along with the five permanent members - the US, the UK, France, China and Russia.
"Work mechanisms and double-standards on the Security Council prevent it from carrying out its duties and assuming its responsibilities in keeping world peace," the Saudi foreign ministry said in a statement.
"Therefore Saudi Arabia... has no other option but to turn down Security Council membership until it is reformed and given the means to accomplish its duties and assume its responsibilities in preserving the world's peace and security," it added.
The failure "to find a solution to the Palestinian cause for 65 years" had led to "numerous wars that have threatened world peace," the foreign ministry said.
It also criticised the UN's "failure" to rid the Middle East region of weapons of mass destruction, including nuclear weapons.
And it accused the UN of allowing the Syrian government "to kill its own people with chemical weapons... without confronting it or imposing any deterrent sanctions".
The UN Security Council last month finally broke a two-and-a-half year deadlock on how to deal with conflict in Syria after voting unanimously to adopt a binding resolution on ridding the country of chemical weapons.
Saudi observers say Riyadh would have been working for years to gain a place on the UN Security Council, so such a decision would have to have been made at the very top of the kingdom's leadership.
One Security Council diplomat, quoted by AFP, said the announcement was "totally unexpected" and without precedent.
Another said: "There was nothing controversial about the election. But the government has made it clear in recent weeks that it is concerned about Syria and the Palestine issue."