As a water heater ages, the tank starts to rust. This rust sits in the bottom of the tank. When you run water, the rust gets into the water. The best way to get rid of this is to open the drain valve of the water heater, and running a hose to a nearby drain. Then you drain the water heater completely. When water stops running you turn the cold valve on while the drain is still open. This will flush out any rust at the bottom of the tank. When the water out of the drain stops being rust colored and clears up, you should be good. Also you may have to run every faucet until the hot clears up.
If you have galvanized piping in the house, it could be rust in the hot water line as well. The only real way to get rid of this is to replace the piping. You will just have to run the hot water until it clears up before every use. It isn't "dirt" just rust from either your piping or the water tank.