Thanks for the answers to my questions although I'm sorry I was off my computer when you responded back.
That's great that they're not otherwise acting ill.
At this age, there's usually going to be either one of two issues which might cause straining or problems with the stools:
1. By far, internal parasites such as round worms are often going to be the problem since they're commonly seen in puppies this young. It should be relatively easy to worm all of them (including mom) with Pyrantal Pamoate which is available at most pet or grain stores. It goes by Strongid, Nemex or Evict (and possibly other names) but the active ingredient is the same. The dosage amount is listed on the label. I tend to worm every two weeks for three doses.
Since this is such a safe wormer, it can be given every week, if necessary, and no harmful side effects would be expected.
2. It's possible that the puppy food you're feeding them is the culprit. If that's the case, then I'd feed what you fed your previous litter but I'd omit the puppy milk (which I assume is milk replacer). They don't really need it at this point and it may also be causing some issues.
I'd also add a daily probiotic such as Forti-Flora which may be found at local pet/grain stores or may be purchased online. Not only can probiotics help with gastrointestinal issues but it can also help to strengthen the immune system.
I hope this helps. Deb