I've heard exFAT doesn't play well with homebrew (hence why I think stuff like Retroarch/Lakka won't ever support it) so maybe try experimenting with FAT32. And I guess you could try playing around with cluster size although not sure which ones would be better or worse for you.