I hate slackware :p Debian and Gentoo work nicely for GUI-less systems for me. I've just finished setting up a headless gentoo box with only SSHD and Samba running on it, and I do everything I need to over SSH to it.
The "screen" package is quite handy in this situation too, since it means I only need a single SSH session/window into the box, yet have multiple bash sessions in the one window. Plus I can disconnect screen and close the SSH session, then SSH in from elsewhere and reconnect screen and continue where I left off - anything running in screen continues running instead of dying if my connection to the machine dies :p