I'm an atheist, and I grew up in the Methodist church, albeit with incredibly decent parents who taught me to love my fellow human beings and judge people based on how they treat others, not their religion, race, ethnicity, or sexual orientation.
I have many problems with Christianity, not least its legacy of bloodshed, anti-Semitism, intolerance, and oppression. On all these counts, Judaism is superior in my eyes, because the oppressed are obviously morally superior to their oppressors. I have never seen Jews trying to make legislation supporting their own religious views and interfering with other people's lives - in the U.S., that is a uniquely Christian (and primarily evangelical) course of action.
I'm currently studying the history of Christianity at Princeton Theological Seminary, or at least auditing courses there, and it has made me very interested in the history of Judaism - it is impossible to understand the origins of Christianity without reference to Judaism. As such, I have been asking a lot of questions here, but I am always afraid of inadvertently causing offense.
This meta question attracted some useful answers, but they are mostly related to mistakes that I, as an atheist, am unlikely to make - for instance, I wouldn't refer to Jesus as "Christ", because I am convinced that he was just a man, nothing more or less. I already understood the issue surrounding the use of "NT" and "OT", so I wouldn't make that mistake either. My interest is historical, not devotional (i.e., I'm not concerned about whether one religion is "right"), so I don't have to worry about showing bias towards one religion. I'm asking about more general guidelines.
Therefore, I have to ask a pretty broad question: What should I keep in mind in order to avoid causing offense? What are the most important things that I should bear in mind?