​

Homebrew and romhacking developers are always working on great things to bring forth new ways to experience great classics for older consoles. One such example is in DS emulation, which has seen it's fair share of attention these past years, like the introduction of a new emulator in the form of melonDS, which seems to be in a fast track to becoming the defacto DS emulator, and today it seems like a big breakthrough in DS emulation has appeared based on the work of melonDS.

Software developer Zhuowei Zhang has developed a proof of concept of an augmented reality feature for DS emulation, in which a holographic 3D image of the current game floats above the controller the player uses for the emulator.

​

As detailed in the tweet, the proof of concept works with melonDS and the melonDS core, utilizing a live 3D model extraction called MelonRipper, and the augmented reality development is possible using iOS' RealityKit. The prototype currently has some downsides, which Zhang goes into detail in the GitHub repository for the project:

• the MelonRipper->RealityKit converter doesn't work very well (e.g. doesn't handle transparency)
• there's a terrible memory leak (might be this?) that crashes the app after a few minutes
• only tested with the camera position used by Mario Kart DS.
  • (for example, Pokemon Black and HeartGold use a different camera angle, and I had to remove the shader that crops the model for anything to show up)
• no way to select rom/touchscreen input/etc. The ROM name is hardcoded to "rom.nds" in the app's folder in Files/iTunes.

Despite these downsides, the project as it stands is already outstanding, giving players some glimpses of the augmented reality with some DS titles like Mario Kart DS, Pokemon Heart Gold and Pokemon Black, giving a floating image of the games with great detail (aside from the transparency issues mentioned in the repository).

The project is currently open source and available in Zhang's GitHub repository linked below.

DSReality GitHub Repository

 

[chrisrlink]

WHOA great job

 

[Aoba]

Holy cow... Love to see this kind of thing.

 

[SuperDan]

That's Friggin Amazing !

 

[Foxi4]

That is pretty damn neat and not at all what I expected - I thought the guy would just map two screens to two QR’s to replicate the physical arrangement on the DS, but this has full on depth to it too. Very cool stuff, if not particularly functional.

 

[KainReaver]

Dont think metroid prime hunters would translate very well to Augmented reality.

 

[SecureBoot]

I don't know why I can't process this in my brain. The footage is taken from an iphone camera which uses the qr code for tracking?

 

[raxadian]

While this is cool, I got tired of AR thanks to Pokemon Go.

 

[Xzi]

Will benefit some games more than others, but very nice indeed. Looking forward to when they can tune out some of that visual noise and stuff that wouldn't actually be visible in-game.

 

[FanNintendo]

Interesting . . .

 

[x65943]

As jank as it looks, the AR pokemon is something I really wish takes off - love the way you can look on like a little miniature set

 

[VashTS]

I don't know why I can't process this in my brain. The footage is taken from an iphone camera which uses the qr code for tracking?

same. confused

 

[Qubarf]

Holy HELL dude - that is one impressive as hell GitHub!
Say hi to Google/NASA/Microsoft for me.

 

[LeyendaV]

The community does what the industry cannot.

 

[M7L7NK7]

I don't know why I can't process this in my brain. The footage is taken from an iphone camera which uses the qr code for tracking?

Do you need glasses or a headset or something? Also confused

 

[Flame]

I'm learning things which I didn't even know was possible. Homebrew scene is amazing.

 

[Ligeia]

Super impressive on a technical level, quite futile from a gameplay perspective.

 

[lenoa]

mario kart vr

 

[Lostbhoy]

I don't get this... Ar on a ds but they using a switch and obviously another camera source??

I dont understand what they are playing on, how they are viewing it, what device im watching in the video and my... it looks god awful... What is the point??

Im not trying to bash it here, just understand the what and why as personally it seems a pointless, overly convoluted way to play a ds game.

Can anyone explain a useful reason for this as I can't think of any ds game I wanna play like this!

 

[bazamuffin]

mario kart vr

Already done but with Mario Kart wii