So here's the thing about Toribash: It's a niche game. There are very, very few games like it, and it follows almost none of the standard formulae. This means that, as Gruszex pointed out, you have to learn everything from scratch, since you can't really draw on any past experience. Picking up a new FPS isn't going to be an issue if you're already a Battlefield veteran and so on, but Toribash? That's like going from drums to violin, it's a completely different skill set.
I will say, however, that Toribash has fared MUCH better than a whole host of other indie games that aren't as unique in gameplay. I'm actually surprised it made it to 4.0, and here we are, past 5.0. Still, i believe the main problem is that it's just not a game for everyone. If you don't fall in love with the concept at first, you most likely won't ever, and if you DO fall in love with it, you might still get turned off by how difficult it is to get started. It's very difficult to do anything about whether people like the concept or not, and as Gruszex ALSO mentioned, it's also very difficult to teach people how to play the game without just saying "practice lol". I mean, like most other games, listing down the controls you use is trivial, but using these controls effectively can only really be learned with a shit ton of practice. It's similar to saying that you play the guitar by plucking the strings while holding chords. It's true, but not even the best teacher can get you proficient in a day.
Long story short, i'm surprised it isn't dead yet. Getting new players and getting them hooked is what's going to be needed, but alas, ragdoll physics aren't as new or amazing as they were 10 years ago, so i'm not sure that particular hook can be relied on any more.