Sure, I can understand you seeking privacy within a private server, but at the end of the day there's more reason to keep the in-game admin for staff than not to. Private servers are still under global rules, and sometimes we need to join said room to change the description, or server name, as it can be offensive/harming. There's been countless cases where a server was full or others were unable to join and needed assistance, removing that would affect greatly those that do actually need it, and the only real upside is for those that simply want to feel the power of lightning bolts shooting from your fingertips as you ban a purple name.
As a sidenote, I would feel more "lightning bolts" when I told someone to get out and make them switch rooms themselves than just press a button to ban ^^