It's pretty straightforward once you have a way to compile with devkitPro set up. I'm not sure how hard that is on a Mac, I use Windows. The Doomhack repository has fairly easy instructions, though. (Gah, I'm new here, can't post links yet... just search for GBADoom on Github)
Once you've got devkitPro set up, download the GBADoom code from the repository and extract it someplace you can easily find with the MSYS shell that comes with devkitPro. I keep my stuff in C:\\devkitPro\projects.
Go inside the GBAWadUtil folder. Drop your IWAD files here. If you're on Windows, you can open up a command line window here (Shift+right click gives you an Open PowerShell option), and run a command like this:
Code:
.\GbaWadUtil.exe -in IwadFileName.wad -cfile CFileName.c
For Freedoom, I used
Code:
.\GbaWadUtil.exe -in freedoom1.wad -cfile freedoom.wad.c
There's ready-made batch files too that you can use. These output straight into the source\iwad folder.
Move the generated .c files into the source\iwad folder (if they're not there already).
Look in the source folder, find doom_iwad.c and open it in a text editor. Comment out (put // before) the "#include iwad/doom1.c". Add a new line in that block that looks like "#include "iwad/CFileName.c"". When I was doing Freedoom, at the end my doom_iwad.c file looked like:
Code:
#pragma GCC optimize ("-O0")
//#include "iwad/doom1.c"
//#include "iwad/doomu.c"
//#include "iwad/doom2.c"
//#include "iwad/tnt.c"
//#include "iwad/plutonia.c"
#include "iwad/freedoom.wad.c"
#include "doom_iwad.h"
const unsigned int doom_iwad_len = sizeof(doom_iwad);
When yours looks similar, close the text editor, navigate the MSYS shell to the top folder (where you can see the build, source, include, etc. folders), and run "make". Let it chug and it should eventually spit out a GBADoom.gba ROM.
Hope this helps!