Ok. If the judge actually issued an order stating that the amount owed was only $5000 (because that was the total amount that the court could issue an order for), then you could not be sued for the remaining $3300.
The concept of res judicata or collateral estoppel simply means that a person may not relitigate an issue that was decided by a court in a judgment.
Now, I have to be clear here. If the court did not issue a judgment specifically stating that you only owed $5000, then the issue was never actually decided by the court.
Let me give you an example. Let's say that the court orders you to go to mediation (which is not court....no decision of law is made there). You agree to pay $100 a month. During that mediation, the mediator does mention that small claims only allows for a maximum of $5000. The plaintiff acknowledges that. You agree, in mediation, to paying $100 a month. The court dismisses the claim.
You pay $100 a month, and end at $5000. If the statute of limitations
has not run, they could still sue you for the remaining $3300. The reason is that, in the scenario that I mentioned, there was no final resolution by the court. The court didn't issue an order, but rather, dismissed the claim because you reached an agreement for payments in the mediation.
So, what really matters, in terms of the answer to the question, is what exactly the court did here. Not the mediator, who has no deciding authority at all. If the judge issued an order, and that order states that you owe $5000, the issue can't be relitigated.