Hi and welcome to Just Answer! I'm happy to help with your tax and finance questions.
Your first step is to determine what type of business entity to use for your business. Since you are first starting out, I suggest operating as a sole proprietorship. This is the least complicated type of entity and doesn't require any additional tax filings or additional maintenance costs, as other options, such as a corporation or an LLC. However, you may want to consider either a corporation or an LLC, as these offer greater personal liability protection than a sole proprietorship.
If you are going to be operating the business under a different legal name (e.g. your name is XXXXX XXXXX business is Green Lawn Care or your corp's name isXXXXX but you go buy Green Lawn Care), you will need to apply for a DBA (doing business as) name. This link from the Small Business Administration (a great resource for information, by the way), discusses DBA filing requirements in the various states.
Typically you can obtain a business license from either your state or local municipality. I can check into the rules for your area if you want to provide me your location. If you are not comfortable doing so, then that is fine too. You may also need to register for state and local taxes.
You will want to obtain a good general business liability insurance and should consider a personal umbrella policy as well to shield yourself against liability. You can contact a local insurance agent for more details on policy costs, coverages, etc.
If you are going to be hiring any employees, you will want to apply for an employer ID number from the IRS. There is a link on the left side of http://www.irs.gov/ to fill out an online application.
This link from the SBA is a good reference to the steps of starting a small business: http://www.sba.gov/content/follow-these-steps-starting-business