Unfortunately, you cannot add previously removed contacts to that same campaign from which they were removed in the first place.
They were not removed completely. In other words, they are part of the statistics.
They are still in the campaign, but their status is changed. This means that they won't receive any further messages.