All the instructions are at the retroleap github. Take a look at the bottom of the page.
clone the retroleap github here:
https://github.com/mac2612/retroleap
Code:
make leapstergs_defconfig
make menuconfig
make menuconfig will build and run a ncurses GUI menu to choose what you want to build. Leave everything alone except for the retroarch cores section. Select the cores you want to build: TG16 PCE, Gambatte, and Stella work pretty well; Wonderswan only so-so.
Save your changes, exit menuconfig and type make. This will take a while to download more stuff and compile: it took over half an hour on my system.
The sources (and binaries, object files, filesystem images, etc.) are about 2GB right after building.
I think I also ran into a few problems compiling...stupid stuff like missing symlinks.
Bilinear filtering ON and Video Threading ON seem to be the best compromise with Gambatte. Actually GPSP doesn't look all that bad with filtering OFF; it's just the fonts that look grubby.
If you don't want to compile this mess, you can
download my sshflash.tar.gz with the filesystem image containing more cores.
About the USB network mess:
I reflashed my LeapsterGS using a manjaro linux system with KDE's networking GUI. I had to disable DHCP and manually set the USB interface's IP to 169.254.8.2 (The leapsterGS itself is 169.254.8.1 .) It still took about 10 tries to actually get a stable connection and have the flash succeed....
After you have reflashed and rebooted, the leapsterGS changes its IP to 169.254.6.1 so I changed my USB interface to 169.254.6.2. (No idea why the IPs have to change or why the silly rsa/ public key bullshit... I think the danger of unauthorized persons breaking into my almost never networked leapsterGS is pretty miniscule. Maybe this firmware was intended for something else originally?)
Anyway, I just used scp from the command line. It accepts wildcards "*.gba" and can recurse subdirectories if you put -r before the -i. There's only a little more than 1 GB storage on the leapsterGS.