Congratulations on your new addition and thank you for taking on a rescue kitty - he is very lucky to have you indeed!
You are definitely right to separate the kitty from your dogs when you aren't there to supervise. It's difficult to truly dog or cat test these critters in the shelter, as sometimes their reactions in the shelter are tempered by the stress of the whole situation and not truly indicative of their true personalities. I trust that these organizations do the best that they can, but sometimes they do get it dead wrong, as I see happened for you. I'm sorry for the extra stress that you're enduring!
If your dogs are quite calm and tolerant of the cat, it's certainly reasonable to give the cat time to settle in and relax. With everything being so brand new just after being released from a stressful situation like a shelter, his true personality is likely not to show yet for a few weeks to a couple of months. If he's never been around dogs before, how he adapts to the dogs depends on how your dogs react to him. What you are seeing in his behavior is likely all based in fear. When cats are afraid, they put on a big show of threatened aggression in hopes that it will scare off the "bad guy" - in this case, your dogs. If you dogs are calm and ignore him or have little interest in him (thus seeming less frightening in the cat's eyes), then he will likely learn to tolerate them. However, if the dogs are more likely to get excited, poke, pounce, or chase him, then the cat is going to continue to be afraid and the situation may even get worse. I see many mixed households where the cat(s) rule the home, and the dogs won't even walk through a doorway if the cat is sitting nearby (my own large Greyhound won't go into his own bed if my tiny cat is sitting there! And she's never even peeped at him before!).
Having said all of that, though, if the kitty really doesn't fit into your home, there is no shame in returning him. I'll tell you, not only are your dogs unhappy if they have a foul-tempered roommate, the kitty is likely to be stressed right now because he's terrified of the dogs and feels in constant threat. If the dogs are calm and accepting, the kitty may get over this in time, however there's no guarantee. If he's not suited for you home, and the shelter is a "no-kill", then he will be put back up for adoption with the stipulation that he only be adopted to a family with no dogs, and he would likely lead a happy life. We don't judge for this sort of return - it's not your fault that they don't get along - you are being responsible and cautious, and that's OK too.
Please let me know what other concerns I can handle for you - we can chat until I've addressed them all.