you can make a handheld with a pi??
Sure, there are dozens (if not hundreds) of case designs and instructions around for making your own Portable Pi. So long as you're half decent at DIY, have moderate soldering skills, and can follow directions you can get it all done yourself sometimes for a little under $150, depending on how you do it/what Pi you end up choosing.
One of my favorite designs is the PiGRRL2, which would be perfect for your requirements and only
costs maybe $110-$120 to do yourself (unless you already own a Pi, that is, and maybe more if you don't have easy access to a 3D printer):
https://learn.adafruit.com/pigrrl-2
The D-pad and buttons will be as good as you want them to be, and it looks close enough to an OG Gameboy that you could probably trick people who don't know any better that it's totally an official Nintendo product (that is, assuming you take care in putting it all together).
The only problem with using a Pi is input lag, generally speaking a stock Pi will hit like anywhere from 8-10 frames of input lag (depending on the game, display, and controller, of course, but this is around the average for the optimal experience without any tweaks). You can, however, reduce that with software tricks (and I imagine using a wired controller in this setup would also help, as well) to around 3-5 frames of input lag which is basically indistinguishable from the real thing at that point.
Assuming you're shit at DIY, or have no soldering skills, I would personally recommend a Vita over the PSP nowadays. They can be had for $100 (most places, anyways), are easy to hack (so long as it's on a firmware under the latest, which is 3.69), you can use Micro SD cards with an SD2Vita, the buttons are quite frankly fantastic, and (as far as I can tell) you get pretty great emulation for 16 bit consoles with little issues thanks to Retroarch at this point. Plus you can play nearly all PS1 and PSP games near perfectly, thanks to Adrenaline. The only thing really missing is N64 emulation, but beyond that it's pretty decent for a portable retro console.