Supporting my (atheist) Jewish husband. Need advice.

My husband (42) and I (38) are newly married with a baby on the way. We will be moving into our new home soon, and I want to surprise him by adding a few Jewish decorations to our home. We are both atheists and he is Jewish. His father went to Hebrew school, but did not incorporate much teaching, religion, or culture into his sons’ upbringing. Possibly because of the antisemitism of our location. Can anyone give me some advice about how to incorporate some items into our home to support his (and our baby’s) heritage? Any advice about learning more would also be appreciated! Thank you!