As someone who's owned both, I'd say the Switch. Despite it being harder to mod than a Vita, I'd say it has quite a few advantages IMO. A larger screen, It's still actively getting games released for it of all sorts of genres (I liked having a Vita for JRPGs but a lot of the time it felt like that's all there was), has more powerful hardware, can emulate a ton of systems fairly well if you can get your hands on a modded system (for example, I've been playing Zelda OoT on RetroArch using mupen64-next, Racing Lagoon on Retroarch using PCSX-Rearmed and GTA: LCS / VCS via PPSSPP lately and apart from the GTA games having a bit of stutter when driving super fast vehicles, all the emulators I've listed are usually a fairly smooth experience), and not to mention a modded system can run Android and Linux, meaning you can run 3 different operating systems on it, and a bunch of other stuff.
Of course it has it's disadvantages (price point, joy-con drift, the e-shop loading times will make your hair turn gray, etc.) but ever since picking up a Switch, I've rarely wished that I still had a Vita except for maybe wanting to play Persona 4 Golden. Also if you want something easily hackable, 3DS systems are pretty cheap these days and incredibly simple to mod.
None of this is to say the Vita's a bad system, it just falls a bit short because of how quickly Sony abandoned it, but the homebrew scene is always doing cool stuff to make up for it. That and memory for the Vita is expensive unless you go the SD2VITA route because Sony loves their proprietary media formats.