It hasn’t really bit me, although I’ve seen it on on .mpg files, where I get little apple filter turds all over the place that makes programs that scan for video files (xbox media center, primarily) crazy.
I don’t have any .ogg files, though, so I never observed that the libogg crashed on badly formatted files.