Very aggressive chihuahua toward men?

I have a 7yo male; despite the fact that he has "papers," and brought his former owner large stud fees he was brought to a no kill shelter 4 years ago. I was contacted bc of his aggression. Love at 1st sight and he has been my constant companion for the past 4 years. My daughter was 8 when he came home with us and he never bit her. BUT it took 3 years for him to let her touch him. I have much experience training dogs and much success. 3 years ago we rescued a 4week old chi from deplorable conditions and close to death. She is thriving and the 2 have always gotten along. I immediately had the male neutered and the female was spayed @ 6 mos. I remarried TWO YEARS AGO and my male is STILL BITING my husband. All my spouse has to do is walk by and he is bitten in the leg or wherever my dog's teeth can reach. I have always been able to extinguish these behaviors with other dogs. Not this one. My spouse is wonderful and says not to worry b/c my dog is 5lbs and cannot do any damage. I do worry b/c something is wrong and has gone on so long. I would so appreciate and be grateful for any and every suggestion. Much Thanks.