When I try to launch the game without dxwnd it works fine, but when I use dxwnd to launch it i get an error "Cannot find 640x480 video mode".

That makes it sound like it's checking a different directory for the game files for some reason, and thus isn't getting the correct resolution from your normal settings file in GTA Vice City User Files. Make sure you aren't using any compatibility mode stuff for either dxwnd or the game, and maybe try different versions of dxwnd or something.