I can understand why you might think that, but no - I too have a decent tv (Sony Bravia) that has extremely low latency in gamemode. My connected retro (ie, not natively HD) systems (Gamecube, Dreamcast, NeoGeoCD, Megadrive/MegaCD, MasterSystem, SNES, N64) have no discernible input delay. The only system connected (out of 15 total systems) that has obvious input delay is the PSTV when using a wirelessly connected dualshock pad. It's especially noticeable when playing anything I've grown used to playing on my PSV1000.
Not saying your TV is worse or mine is better, just that the PSTV (in my experience) has the worst input delay I've experienced on any system in my setup. It's still an absolutely awesome device, but I tend to only play titles on mine that don't require the same precise timing of inputs that things like fighting games do.
It performs as expected using either my wired Hori Fighting Commader Pro or Madcatz Fightpad (but the fact that they're wired ends up with them dragging the tiny PSTV all over the room) - it's just the dualshocks that have the lag.