It might come as a surprise to you but no, not all impacted wisdom teeth necessarily need to be removed. It certainly can be possible that a person with an impacted wisdom tooth could live their entire life without ever experiencing any problem with it.
Of course, the difficulty associated with evaluating wisdom teeth is being able to determine which ones are likely to place the person at risk. This is where your dentist's experience and judgement comes into play.
What type of impacted wisdom tooth is most likely to place a person at risk for experiencing problems?
Any retained wisdom teeth should be monitored.
Over time a wisdom tooth's status may change. This includes both the potential for the formation of pathology or a change in the tooth's positioning (i.e. a full bony impaction becomes a less predictable partially erupted tooth).
For this reason, any wisdom tooth, but especially an impacted one, should be routinely evaluated both visually and radiographically (with x-rays) on a schedule suggested by the person's dentist.
Some "impacted" wisdom teeth probably shouldn't be considered true impactions.
It's important to keep a wisdom tooth's normal eruption process in mind when passing judgement on it. A typical time frame for third molars to erupt is between the ages of 16 and 25 years.
For people in this age range, in some situations, a case might be made that it's debatable that the term "impacted" really applies to their tooth. After all, it's quite possible that the tooth is still erupting and has not yet arrived at its final positioning.
That's not to imply that it isn't sometimes clearly obvious, even at a young age, that a wisdom tooth will never be able to erupt successfully. But in those cases where available jaw space and the tooth's alignment appear to be somewhat within normal limits then quite possibly the tooth should be given some leeway, in the sense that it is given more time to progress with its eruption.
Then, at an age lying closer to the end of the normal eruption time frame, if it appears that the tooth will be incapable of erupting appropriately, it can still be extracted within that time window typically considered best for extracting wisdom teeth (the upper boundary of which is on the order of age 24 years).