Have Parenting Questions? Ask a Parenting Counselor for Answers
I'm a polyglot myself. Because you he's to be interacting with English speaking children and teachers, you would want to stress the English language first as his primary language.
Then, you being the primary caregiver and his mother can tech him your native language.
When he is older, he can still pick up on his fathers language once he has had the foundation of these two languages.
You may consider which language is more widely used- Mandarin or Cantonese and how much he could practice either language with native speakers that are not always family members. As long as there is someone for him to speak with on a regular basis, he can initially start with the English and Cantonese while he's still young. As he gets older, you can introduce him to Mandarin.