It's too bad you would deny your kids their grandfather but I suppose that's your right as a parent. Lots of people do that for far more benign reasons than yours.
Around 12,000 kids die each year and according to the CDC, they're most likely to die in some motor vehicle related accident. After that, drowning, suffocation, "other", fire, ... . 9.2 million visit the ER for non-fatal injuries-falls and being struck by or impacting objects.
My point is having a discussion with your dad about how he stores his gun would be a far better course of action than threatening him with never seeing his grandkids face to face again especially given your kids have a better chance of being killed on the drive over to his house.