it means he is not happy in that situation.
Actually, if the horse's ears are slanted back or slightly slanted (pointing back), it could mean that it is tired, ill, listening, or content.
If the horse's ears are pinned (resting flat against the neck), it could mean that it is angry or fearful.
However, you cannot tell how a horse feels just by its ears. There are many other factors you have to look at.
I have a mare who have her ears back all the time, but she is not angry, mean, or fearful. She usually lays her ears back when she is resting or when I put on the saddle.