Please keep in mind - the scope of this forum is a "general" legal information consultation - so we cannot go through a detailed evaluation of individual claims you have to try to unravel your partnership here.
But in general, you have 2 basic claims in this kind of case: breach of contract, and breach of fiduciary duty.
These two causes of action will generally be sufficient vehicles to allow you to get into court and to recover whatever substantive damages that you are able to prove.
The more complicated part of your matter deals with how you are actually able to divide your interests/what you are claiming for damages/etc. AND, more importantly, have you already waived a right to any of these by agreeing to accept back your initial investment and walk away?
Generally oral partnership agreements are very difficult to manage because partners contribute very different amounts of effort and/or capital, but the court will only be able to divide the partnership according to how many partners there are unless there is a clear indication that one or more partners was to be reimbursed in a greater percentage.
If you can show that your initial investment was to be reimbursed as a loan to the partnership, instead of as an investment in the partnership - you may be entitled to recover this in addition to your percentage of the partnership profits/assets at dissolution.
If you cannot show this (your investment was really an investment - this would be the default), you would only be entitled to a return of your portion of the partnership at dissolution.
As you are not dissolving the partnership, rather you are withdrawing as a partner (the difference is that the business is continuing on without you), it makes it very difficult to properly value the business for dissolution purposes (a complete liquidation of assets, payoff of liabilities, and distribution of any remaining capital).
What your right would be as a partner desiring to exit the partnership would be to sue for dissolution (forcing the others to close the business and dissolve so that you can be paid out - they don't want to do this, and neither do you, as it devalues your business - so they pay you off for an agreed upon amount). Your business partner agreed to do this already.
However, you rejected this offer, and agreed to accept his return of your investment only. If you have done this it is going to be difficult to get a new offer back on the table. I don't know any of the individuals or personalities involved, you may have an easier time than others might in similar situations. You can also try using a mediator to step into this matter and help you work up a proper exit contract that relieves you of liability from the partnership.
In the future - make sure to get a proper partnership agreement (as drafted by an attorney) as this can help you avoid most of these problems - all of these issues, and many more, will be identified in advance along with mechanisms to resolve them (helping to avoid confusion, frustration, and ambiguity).