Hi, I'd like to help you with your question.
I think you have already reached the point where you can be sure there is no healthy relationship here. People who have personality disorders, especially narcissism, have a very difficult time with insight. They simply do not see themselves at fault for anything and everyone else is to blame. Narcissists see the world very much from their own perspective. Everything is about them, even if it isn't in reality. And it is other people who have flaws. The narcissist cannot cause problems because there is nothing wrong with them.
The only way the relationship between you and your brother could change is if he would gain enough insight to see that he has problems that cause you pain. A therapist could help him but he has to want help in the first place to be able to take advantage of a therapist's help. Talking to him or trying to convince him that he has issues that warrant help will most likely do nothing.
What you can do is protect yourself. Staying away from him as much as possible is a good plan. The more you are away from him, the better. He is going to be mad you stay away because he does not understand why. He feels it is you that is causing the problem. And if you would do what he wants you to do (i.e. pay attention to him, give him what he wants) everything would be fine. But in reality, what would be fine is him, not you.
You can also develop a few responses you are willing to have with him. Make them simple, and unemotional. So when he accuses you of not paying him enough attention, you can say to him, "I'm sorry you feel that way" then walk away. Repeat as necessary. He may get upset but make a plan to stay away from him or bring someone with you who can help you if needed. By doing this, you can keep yourself calm and also give him nothing to attack or hurt you with. It also gives you some control in an uncontrollable situation.
Here are some resources to help you learn more about your brother. The more you know, the better you can help yourself:
The Wizard of Oz and Other Narcissists: Coping with the One-Way Relationship in Work, Love, and Family by Eleanor D. Payson
Disarming the Narcissist: Surviving and Thriving with the Self-Absorbed by Wendy T. Behary
The Object of My Affection Is in My Reflection: Coping with Narcissists by Rokelle Lerner
You can find these books on Amazon.com or your local library may have them for you.
I hope this has helped you,