you just sit there and keep yourself occupied until everybody catches up
That is just wrong on so many levels. In the late 1980s when the disability community in the United States was fighting for legislation that would guarantee appropriate education for students with disabilities, they reached out to the gifted community to join the fight. The gifted community balked at the idea, which is why individualized education plans are a legal requirement for students with disabilities but not for gifted children. Which sadly means that a school today could get away with doing what your school inflicted on you 45 years ago. Our maths panel this weekend will be touching on this to an extent (my cheap plug), but in some cases it really comes down to a need for legislation to provide a basic bill of rights for gifted students.