Hi this is Dr M and I would try to assist you today. Decrease in activity level can be caused by many different things. Some would be less worrisome. As a ferret gets older, there will be a decrease in activity level. Usually, they sleep more but when they will still have good playing times as before. Some diseases can be important and make a ferret more lethargic like insulinoma. Insulinoma is a very common disease in older ferrets (greater than 3 years of age). The pancreas over produces insulin that decreases blood sugar levels and therefore, decreasing levels of the internal fuel for the body. A blood test would be needed to look into this type of disease. Another possibility could be heart disease. Do you think this panting you mentioned matches the beginning of this increase in lethargy? If so, heart disease should be also considered.
These are just some suggestions. Please let me know if you have any other questions.