He doesn’t panic, even when you expect it, because he loves you.
It is completely acceptable that for some people the presence of skeletons in the closet can be a hindrance, because everyone has them. However, if your partner is still with you after learning a lot about your past, even incidents you’re not particularly proud of, they may be more interested in you than you think. Mature relationships and true love show that you accept the whole person, not just the good parts.