Guinea Pigs are pack animals... Visiting isn't enough.. Be wants to be with his family, and chances are no matter how much attention you give him, it won't be enough. As soon as her baby has gone, she'll be unhappy too. Unfortunately, at this point your best option is to have the male neutered (Spaying a female is generally considered a riskier surgical procedure than neutering a male. It is more invasive).
My advice to you would be to find a good "exotic" or small animal vet who has experience with neutering guinea pigs... It's not the same as neutering a cat or dog, so you want a specialist... Call a rescue organization near you (even if it's a good distance) and ask for a recommendation... Many rescues routinely have their cavies neutered prior to adoption and know of good vets in their area and may know of vets in the larger region or state that can help you.
