You can make the textures with any resolution.
When you upload the textures in the
Textures page in Shop, they'll get resized automatically to 128x128, which means they won't be as good quality as you might've previewed earlier.
If you want to keep the high resolution, you just have to save the texture(s) with .tga filetype and then put it in your custom folder.
Your custom folder can be found in ...Toribash-3.88\custom\your_name
Also, you have to save the picture with the correct name.
Those would be...
background.tga (GUI background)
breast.tga
chest.tga
dq.tga
flame.tga (Flame Particle Texture, not the flame itself)
groin.tga
ground.tga
head.tga
header.tga (GUI header)
icon.tga (icon that shows under your name. Exceedingly rare)
l_biceps.tga
l_foot.tga
l_hand.tga
l_leg.tga
l_pecs.tga
l_thigh.tga
l_triceps.tga
logo.tga (GUI logo)
r_biceps.tga
r_foot.tga
r_hand.tga
r_leg.tga
r_pecs.tga
r_thigh.tga
r_triceps.tga
splatt1.tga (GUI splatter)
stomach