He most probably will not hurt the baby bunnies, but the mother may hurt him because she wants to protect her babies. The mother can get pregnant even on the same day she gave birth, so if you don't want anymore baby rabbits to take care of, you should bring the father to a rabbit-savvy vet to get neutered asap. I would suggest that you 1) give the father his own cage or area but in a place where he can still see the mother; 2) get him neutered; 3) let him spend time again with the mother approximately 2 weeks after he gets neutered.
