You should be worried that your SM doesn’t seem to respect your time. You weren’t scheduled that day, so you weren’t obliged to work. Obviously, stuff happens—people call out, a delivery comes in late, etc….and you should be understanding to that as well. But, again, you weren’t scheduled to work, so you have every right to make plans on your days off and have every right to refuse to come in on your day off. That’s why SM make schedules 3 weeks out—so you can plan your time around work.

You're not obligated to come in on your day off! And the fact that they are making you feel guilty for it is crap.

If they ask you to step down, 1 you did nothing wrong and 2 I wouldn’t sit there and let them run you around. If they want to demote you they can run the paper Trail. Don’t make it easy on them to say you “stepped down”