This question should be answered based on where you decide to store your trailer
during the winter months. If it's stored in a damp environment, then yes definitely remove it.
If it is in a heated garage, there is no need to worry about it. Everyone has a different place they keep their teardrop stored. I leave my mattress inside the trailer all year round. However, My teardrop is stored inside my shop and the shop is temperature controlled all season. When in doubt, pull it out :-)