Hi and sorry to hear about the problem your goldfish is encountering.
The condition your fish has is called Swim Bladder Disorder (sbd) which is obviously a disorder that affects a fish's swim bladder. The swim bladder is responsible for a fish being able to swim upright and maintain its buoyancy. When it becomes blocked, infected or too full of air it ceases to function properly as you are witnessing with your fish.
You did good by feeding the fish peas as this is generally the first course of action taken when sbd appears. Feeding flakes on the other hand, unless they're presoaked, is usually not advisable as the fish is likely to not only take in food but also air thus further complicating the problem.
As you've already tried feeding peas to no avail what I suggest you do is try a fast/feed diet as this will sometimes evoke a cure. This type of diet would consist of fasting the fish for two days, then feeding the fish exclusively peas for two days and then fasting again for two days, etc.,etc. This plan should be carried out for 14 days.
In addition to the fast/feed diet it would be helpful to administer a bacterial medicine designed to treat sbd as many cases are either caused or aggravated by bacteria.
Listed below are links to two different medicines that can be used. The first one is more easily found in the USA and the other outside of the USA.
Also, if possible, it would be better to treat the fish in a fish tank aka hospital tank.
Please note that goldfish in particular are quite susceptible to sbd mainly because of the odd shapes of some species and sometimes because of genetics and poor breeding practices, i.e., being inbred too much.
Sbd is truly one of the more pestering problems facing goldfish keepers.
The chances of a fish recovering are 50/50.
Best wishes on a full and swift recovery for your goldfish and please let me know if you have any questions.
http://www.drsfostersmith.com/product/prod_display.cfm?pcatid=4842
http://www.edirectory.co.uk/pf/880/mia/d/interpet+tropical+fish+medicines+interpet+aquarium+13+swimbladder/pid/5166422
.
YOUR TROPICAL FISH RESOURCE
OVER 20 YEARS EXPERIENCE W/TROPICAL FISH---BREEDING, MAINTAINING, CURING