The question is whether the summerhouse formed part of the subject matter of the contract, if it is then you would be obligated to both return it and erect it. If it is not then you would not be obligated to return it at all - there is no halfway house of returning but not erecting (unless agreed by way of compromise).
If the summerhouse has been included in the agents particulars and you have replied "yes" to a question similar to "Is everything listed in the agent's particulars included in the sale" then it would be included in the sale, regardless of your inadvertant answer because if you were not sure it was your responsibility to clarify with your conveyancer.
If the question is something different which does not seek to include the agent's particulars then it is more of a grey area - the agent's particulars are "subject to contract" and it would not be automatically binding and the question would become what was the expectation of both parties - the particulars would provide circstantial evidence in favour of the buyer's position.
If the position as immediately above then I would ask your conveyancer to dispute in correspondence the matter and say that given the lack of clarity over whether it was included that although you are willing to return to avoid litigation you are not willing to affect it being erected.
They may or may not sue - it depends on how much it costs to erect - but you should involve your conveyancer more
If this is useful please kindly click accept so that I may be rewarded for my time. It will be gratefully received and you will be free to ask follow up questions.