(This is based on the average consumer)
The main problem with a portable XBOX console is that the average person would demand/expect something the same size as a Switch, but with specs that match the XBOX One X, as those are now the precedents Nintendo and Microsoft have set. Many people would be very disappointed that the console's memory would most-likely be flash/eMMC, as many users are accustomed to manually upgrading their hard drives since the 360. An HDD or SSD would be unfeasible without adding major bulk to the system, given their thickness compared to the average tablet. Currently, eMMC is up to 256/512 GB, which is nowhere near 1 or 2 TB, so the size is lacking for a system that has 40GB+ games. What would gamers think of an Xbox that requires an SD card or USB drive to have more than 10 digital games and DLC? Also, how much would a 512GB/1TB eMMC drive the price up?
Unlike Nintendo, almost every game on the XBOX has a heavy reliance on online multiplayer, so 4G LTE would be required to play online while on the go. People like convenience, so they wouldn't want to be hassled with setting up mobile hotspot or tethering, which also assumes their phone can do that. Now the system would require a data plan on top of Xbox Live. After all, people don't want to lose features that they have been given on previous console generations. Even at home, many Xbox players use wired Ethernet for better connections to game servers, so it will be really awkward to have a portable system that is constrained by the length of the cord. (This is assuming the console is strictly portable and not a hybrid like the Switch)
I could go on about 4K, 5+ hour battery life, and other things, but simply put, Microsoft would have a HUGE challenge developing a portable/hybrid console that could be priced competitively with the Switch that meets the demands that Xbox users would expect.