Hi, I'd like to help you with your question.
It does sound like this could be one of two things. Either your husband is considering an affair, or he is using the idea of an affair as a threat to control you.
From your description of his behavior, it does sound like he is using the idea of an affair as a threat. You mentioned that he tries to control other areas of your life like how much you work and by coming home yelling at you and criticizing you. These are all indications that he is trying to control you.
Somewhere along the line, he picked up that you find the idea of him having an affair upsetting. Maybe you have talked about it or it came up in an argument. Either way, he knows this information and is now using it against you.
There is also a small possibility that he is really having an affair. There may be signs you can look for to tell if he is cheating. Has his behavior changed a lot recently? Does he take more care in grooming or has he lost weight? Has he stopped coming home during the week or been gone more over the weekend? These are some of the changes in behavior to look out for.
Do you feel your husband would be willing to go to counseling? The issues in your marriage need to be addressed and his treatment of you should change. He needs to learn better ways to communicate and how to deal with his anger issues.
What he is doing to you is a form of emotional abuse. There is also a site that can help you learn more about his behavior and what you can do to help yourself:
Are there any friends or family you can turn to for support? It is important that you have others you can turn to if you need extra support or if the situation gets worse.
You also might want to consider seeing a counselor if your husband refuses. Talk with your doctor about a referral or if you attend church, your pastor can help. You can also search on line at http://therapists.psychologytoday.com/rms/. A therapist can help you sort out your feelings and determine how you can respond better to your husband's issues.
I hope this helps you,