Well, that's a bad choice because as far as I know, there's no N64 core for 3ds.
But let's put an example with Super Mario RPG: Legend of the Seven Stars for Super Nintendo, just to keep on the Mario RPG thing.
First, you need RetroArch. Currently the latest version is 1.16, but it has an audio glitch on the 3DS version, so I recommend 1.15. Here it is:
http://buildbot.libretro.com/stable/1.15.0/nintendo/3ds/
Grab the CIA version. Once dowloaded, you open the package and see two folders: cia, and retroarch.
The first one only holds retroarch.cia, and can be put wherever you put your cia files to install. The retroarch folder goes into the root of your SD card.
Once it's there, go to the retroarch/cores folder. That long list of files are the emulators you'll use, known as cores. As I mentioned, is best to only use the ones you want to use. In this example we will only use snes9x2002_libretro.cia, so if you want, create a new folder there named Unused, and move all the others there.
Once that's done, go to the root of your sd card, and create another folder, name it whatever you want, and it will be used to hold the games you're going to use with retroarch. I personally order them by systems, and inside each system folder, by genres. This keeps things much tiddier.
Once you have your game, and the retroarch files, you can take the SD card on your 3DS or 2DS (New systems are preferrable, though!).
Once on your console, open FBI, and install retroarch.cia. This will install the main Retroarch access. Open it up, and after a few seconds, you will find yourself in the main menu of Retroarch.
Here, you should go to Load Content (Content meaning games), choose Explore and go to the folder you set up for your games, and choose Super Mario RPG.
Retroarch will, or could prompt you with choosing which emulator to play it on, if you have several of them that can accept the formats that SNES games use. If you have only snes9x_2002 though, it might pick it automatically.
Whatever the case, choose that one if you need to, and then it will be installed in your console.
Wait a couple seconds and you'll see the game boot up and start playing.
And there you go, you have the game running! Now, as I mentioned, I would recommend setting up at least the aspect ratio so your games look pixel perfect.
You can tap the lower screen to enter Retroarch Menu at any time. Doing so will take you to the Quick Menu that I mentioned before. Press B, to go to the Main Menu. Choose settings, and then Video and Scaling.
There, go to Custom Aspect Ratio Width and set it to 256, and Height to 224. The two options above serve you to center the screen if you want it in any particular position. Also set Crop Overscan ON if it's not.
Now, press B to exit back to Settings. Go to Directory. Go to Downloads, the second option and press A. Navigate to the folder you put for the games, and choose <Use This Directory>. Now whenever you choose Load Content, you can go to the "Downloads" and get instantly to your folder instead of navigating to it every time.
Lastly, press B to go back to the Main Menu, and then choose Quick Menu. Press A, and then on the bottom go to Overrides. Then choose Save Core Overrides, and that should be the most basic setup.
After that you can just play. By doing this you've basically set up your SNES emulator to look good, and you can boot any SNES game with that configuration so you don't need to do anything more.
There's a lot more you can tweak but that's the simplest of steps. Borders, core settings to change palettes for systems that allow it, customize controls per game if it's your thing, and a lot more. But you don't need to learn all of that instantly, is better to keep trying stuff little by little if you care about it. I myself I found out earlier that you can apparently change the bottom screen's menu and make it nicer to look at.
Of course anyone reading this needs to understand we're dealing with the system's limitations, and that some emulators might have performance issues for that, while others work poorly, like the Virtual Boy one, but that's more on the emulator itself than the system (it also works poorly on the Wii, but at least that console has the WiiMednafen VB emulator).
It's a matter of trying and finding which ones perform well.