I have an idea: try removing all of the babbles from the bag, give them to two or three monsters to hold, then leave those monsters at the Monstercare. Since you won't be holding any Babbles with you at the time, it is possible he might take note only of the Gold Orb.
I remember I exploited...