Anal swelling in dogs can be due to variety of factors. Your question would have been complete if you also report other signs of illnesses that your dog is having. Anyhow, anal swelling in dogs can be due to anal sac disease that includes
Anal gland impaction
Anal gland tumors
Anal gland infection
Anal gland abscess
Rectal prolapse in dogs also causes swelling of anus. Exact cause of this condition is not known but intestinal parasites can be contributing factor. So, whatever may the cause, take your dog to vet for diagnosis and treatment.
Inflammation of the rectum of dogs can be due to rectal polyps, rectal prolapse or anal sac disease.
In rectal prolapse, rectal tissues protrudes through anus. Exact cause of this condition is not known but it is considered that gastrointestinal parasites can contribute in the development of this disorder. Rectal prolapse requires immediate treatment.
Another possibility is anal glands disease. These anal glands are expressed during defecation. Some conditions can cause swelling in these glands like infection of anal glands or abscess formation, impaction, and tumors of anal glands. Diagnosis is by physical and rectal examinations and blood tests. Antibiotics are used to treat infections and if no infection then only expressing the anal glands will solve the problem. Tumors are treated by removal of affected anal gland.
There can be few reasons of anal swelling in dogs. Most commonly, it can be anal sac disease. Anal glands contain some sort of secretions that are released at the time of passing stools. Sometimes, these secretions become highly viscous and unable to release. This condition can leads to impaction. Sometimes, tumors and infection can develop in these anal glands. All these conditions can lead to swelling of of anus. If you know how to express anal glands then do it know, otherwise take him to vet. If there is infection then use of antibiotics can be helpful. Anal gland tumors are removed by surgery.
Other possibility of anal swelling in your dog is anal or rectal prolapse. Etiology of this condition is not known but intestinal parasites have some role. This condition is also treated by vet. So, it will be good to get opinion from your local vet.