Finding crashes is a start.
Most systems are not all that closed -- even if they are not open sourced/source released in any useful way they will likely still use libraries for images, web browsing, video decoding... and those will be open source after a fashion, or at least have public bug trackers. Many times said trackers will lag behind things (see also the endless discussions about responsible disclosure) but ultimately show what is up. Most console devs don't seem to update or use current libraries so you then find out there.
The systems themselves might not be closed either -- if Intel somehow pulled a coup and got the contract to make the next xbox then their PR wing would be shouting about it (as might their reports, logos on the chips, possibly an e3 presentation from Microsoft trying to get fanboys slathering with tech talk they don't understand) but also as Intel are not going to design anything truly new (being Microsoft the whole point is that is is basically a windows PC you make your games for) you can probably look at what is already available and move sideways. Said chip might have a particular flaw you can exploit.
Should all that not prove useful you can start "fuzzing" inputs. If a dev does not check their inputs (or expects the limited input that the console provides to be their check) then that provides an in, or if you prefer there is a reason we have a thousand different save hacks, custom level loading hacks, load this web page hacks or even microphone hacks (
https://hackaday.com/2014/12/31/running-nintendo-ds-unsigned-code-with-audio/ ) for various systems. Not as useful in the modern world as things are increasingly sandboxed or signed for a specific console but that does not mean you skip it entirely.
Even without that you can still manually go through the system. For a modern system it might take a while to get a clean dump (be it from memory, because the encryption used, because you need to find a beta/dev/unmodified/specific version/repair tools, because it is really secure if you are not a nation state or high end university*...) but once you are there you can start pulling it apart. I mentioned shoulders of giants earlier -- some people really really like messing with one aspect of low level code, or are really good at power analysis/sidechannel attacks, others might be really good at ROP... again there is a reason why if you watch a hacker conference talk on a new console that multiple people will be there taking individual aspects. That might not get you all the way in but you can then publish what you have and someone else can run with it, or you can pick up from where someone else left off. Even narrowing down things but saying what they are not can be helpful, just don't believe people too hard -- there was a
fairly noted talk about the 360 security system once with a comment along the lines of "this particular bus has nothing interesting on it", said bus a couple of years later would give us the RGH family of hacks.
If you can then when developers patch systems "for stability and security" purposes then the changes made between them are likely just that. Figure out what was changed and then on older versions you might be able to exploit something, or indeed maybe the newer versions too if they did not do a proper job of fixing the issue.
*guess where there are a lot of bored people with high end skills.