The reason the shirt looks odd is that a realistic shirt has both elasticity and friction, which I would imagine yours lacks.
If a person rotated in a circle, their shirt wouldn't move that much, largely because friction would hold it in place and what little motion did occur would be obscured by stretching.
Your shirt however can only compress, by falling onto itself, and does that very easily sliding over the smooth joints of tori.