I can definitely relate to that. Even though I swam in the ocean before and have put some good yardage in the swimming pool, open water swims were scary. So much adrenaline, limbs splashing everywhere and swimmers fighting for a good line, it is common accidentally to bump into someone, get elbowed in the head or even get your goggles knocked off. All that doesn't help relieve the anxiety. My recommendation is start at the back and just focus on getting into your own rhythm. The goal here is to get over the fear.