Some politicians don’t understand that the Constitution covers freedom of religion but also freedom from religion. (I’m looking at you, Rick Perry.)
There are two parts to the 1st amendment. The Free Exercise Clause allows you to worship in your own way (well, within the laws that apply to everyone). But there’s also the Establishment Clause, which says that the government cannot promote religion or favor one religion over another. 
This is the one that many Christians ignore. “Well, they don’t mean our religion!” they argue illogically. They think that it’s perfectly fine to have government promote their religion and their religion’s laws concerning, say, abortion or gay marriage, but scream angrily at any thought of someone else’s religion being imposed through our laws (or no religion at all).
Lately, other religions (and non-religions) are demanding equal rights, as the Constitution provides. If a Christian group is allowed to put a monument to the Ten Commandments up in a public park, then the atheists cannot be denied the right to place their own monument up in the same park.
The latest incident is occurring in Oklahoma, where both Satanists and Hindus are demanding equal space on the State House grounds after the announcement of a huge Ten Commandments display. This has astounded some politicians who have never read the Constitution.
Oklahoma legislators are outraged at this. “This is a Christian nation and Oklahoma was founded on Christianity!” they say, ignoring the fact that (a) the United States specifically was not founded on Christianity and (b) Oklahoma specifically was formed as an American Indian state (well, at first…).
If you don’t want to see satanist or atheist displays on our public property then the obvious solution is to stop placing your own religious monuments there. The Establishment Clause requires that the other beliefs get equal time. Instead, use the money that would have been spent on this monument in other ways — you know, feed the poor or establish an orphanage. Wouldn’t that be the Christian thing to do?
And best yet, because of the Free Exercise clause, the government would not be able to do anything about it.