It's a really good question and I think it depends on what you're shooting and how small you need to go.
I think the biggest question is how much zoom you need. In my opinion, that's the thing phones still don't do well at all. Some of the photos I've seen you take that were really stunning were action: skiing, up close, with a long lens and many frames per second. Knowing you and your kiting, skiing, and mountain biking habits, it sounds like you might want a long zoom. Also, people who go to Africa want long lenses to catch the lions in the distance.
For a superb and pretty small camera, it's hard to beat the Sony RX100V. And the lens! So sharp. And damn it shoots fast. It's only a 3.5x zoom, however.
If you want a really long zoom, you might consider the Panasonic Lumix ZS100. It shoots slower, but still 10 frames/second, which is pretty fast. But it has a 10x zoom and still an amazing lens.
Let us know what you get and pics or it didn't happen. 📸