It's not referred to as custom firmware on that kind of computer, the actual firmware is low-level enough that it rarely needs much in the way of tweaking, but the whole operating system that runs on top of it, OS X, can be endlessly tweaked and customised however you want. Check which version you're running (Google says OS X Leopard is the latest), and search for tweaks and customisation for that. It's an old version but all the info and software will still be out there.