With your family history then a colonoscopy at age 40 is a good idea. If everything appears normal then you will not need any additional screening until age 50, however if there are any polyps present then these can be sampled and analyzed. Depending on this finding then you may have more regular follow up.
Aside from bowel cancers, your doctor may also ask about family history of breast and ovarian cancer as there is a potential link with one genetic syndrome.
Did anyone ever do any testing or give you any names of any conditions that your father might have had?
When it comes to colorectal cancer then gender is not important. We get concerned when people have early breast cancer (before age 40) or ovarian cancer as there is a link to Lynch Syndrome however if the breast cancer was late onset then that is less likely to be linked.
It is good that your brother had no problems as what we are worried about is a condition called Familial Adenomatous Polyposis which causes hundreds of polyps and is associated with high risk of cancer. I would still suggest that you have a colonoscopy in the coming months as that would give you a baseline to know that everything was clear.