I found the issue, and boy did I never think this could cause it. It was the fucking 3V Battery, it was dead at 0V, replaced it, and it works like factory... Honestly, how in the world would it cause that? I thought that maybe a cap was dead, and...
Eventhough the New 3DS XL is more powerful, I still feel like the DS Lite was a more polished system. It's a real shame that it never got an XL variant keeping the GBA slot. You'd have to go on AliExpress and buy an ML shell to give a DS phat the unofficial "DS Lite" treatment, and that's the best we'll ever get I'm afraid.