When I was much younger, and beginning to dive a few shipwrecks in the Great Lakes, I, casually, thought about giving cave diving a whirl - I had a bit of underwater experience and, like you , thought how hard could it be?? As I began think about exploring underwater caves I did some very simple cavern diving in Ginnie Spring Florida and I began to understand just how dangerous cave diving can be to folks who glibly underestimate it - for several reasons. (In cavern diving one can always see the opening of the cave to open water.)
The obvious first reason is that you cannot rush to the surface if you lose your airsupply for any reason - empty tank, failed regulator, dropped mouth piece etc. Ok - redundancy counts, and you can take back up air supplies along - but now you have more stuff to deal with that your life may depend on. A second major issue is that frequently one cannot see even with lights that work due to the turbidity in the water being stirred up by the divers fins interacting with sediment on the cave floor. It is very easy to find you self in zero visiblity even with working headlights. If your lights fails, and the cave goes absolutely black, the claustrophobia and fear can be very intense. Now add the real claustrophobia of narrow, constricting passages with the need for redundant airtanks and breathing systems, redundant lights, and narrow passages and it can get over whelming.
The number of skilled divers who have died as a result of under estimating the risks of cave diving is quite siginificant.
One other factor, is that most sport divers dive in pairs, but you cannot buddy breath in a narrow cavern - you are on your own. That is the reason for the safety line through the cave - so that even in darkness or zero visibility, the trained cave diver should always be able to tell where they are, and which way they need to go.
One of the Thai Navy Seals died in the cave while putting in extra air tanks to help remove the Scouts - no one mentioned that no matter how good Navy Seal divers are, I don't think they actually certified cave divers - I dont know what happened to that diver, but I would bet he had issues with turbidity and lack of visiblity rather than just running out of air. Most divers get comfortable fliping their tank over their head to go through narrow passagways, but usually they do that in decent visibility.
Imagine if one of the yougsters panicked in one of those narrow passages on egress, how would the rescuers deal with that situation, say if a masked was pulled off in panic, since the passageway was too narrow to allow a rescuer to easily replace a mask. I believe the scouts were mildly sedated, but I dont know that for a fact. ( Edit added later - they were in fact anesthetized with some form of drug - I would suspect something like diazepam or propofol, not an inhalational agent. Either given orally, or more likely IV push by one of the Australian divers who also was a practising anesthesiologist )
Or just try this - paint your SCUBA mask with black paint, put it on, descend a dozen feet in a swimming pool, and take all your gear off in the bottom of the pool including fins, tanks, boyancy vest, whatever - and then try to put it all back on with your eyes closed inside the blackened mask. Maybe have a friend move your gear around abit to act like a bit of water flow while you are trying to re-gear. Now imagine, you are stuck in a cave, you don't know which way is the surface, and describe how you handle the situation.
After a bit of thought, I decided to truly admire cave divers and their self discipline, but to not join them. One has to know ones limits, and for me, I decided to limit myself to caverns and an occasional wooden shipwreck.
The professionalism of the rescue effort in Thailand was truly incredible. But it was a pair of British cave divers who found the youngsters, if I am not mistaken.
Here's another link to a TED talk about cave diving - and a partners death while cave diving - https://youtu.be/_EsuHf3_OKg