It could be. You'd need to set up a bridge or ip forwarding system from Windows, and configure Linux to use the connection. I don't know if the Windows binary of VirtualBox is compiled with support for VRDP serving (or USB support, for that matter). The GPL'd version doesn't have either.
With Windows as the guest, you configure it not to show the desktop via the registry; I'm sure it's possible to do that in Linux, but you have to figure out how (and it would vary between DEs). Maybe take out the desktop program from whatever rc script starts the DE services, or just use a WM like flux without a desktop component.
You'd also need to be RDPing into Linux, and I couldn't say how that would work. I don't know of any servers for Linux or clients for Windows.