The most common cause of what you describe is loss of fuel pressure.
The fuel pump in the tank supplies pressurized fuel through the fuel line up to the fuel rail on the engine. When you shut the engine off, this fuel line and rail must retain pressurized fuel so that the next time you start the engine there will be fuel readily available.
What often causes hard starting after the engine has been sitting for a few hours is loss of this fuel pressure; if this happens the engine will need to be cranked several times (since the fuel pump only runs for about 2 seconds during cranking and then shuts off) before there will be sufficient fuel pressure to allow starting.
The usual direction of pressure loss is back into the tank through a worn fuel pump or fuel pressure regulator, although a leaking fuel injector can allow fuel to drip into the engine after shutdown.
To determine if this is the cause of your problem, what you need to do is to connect a fuel pressure gauge to the system, and after shutting the engine off monitor the fuel pressure to see if it is bleeding down. If so, then you need to determine where it is going, by clamping the fuel line and the return line closed to see if the pressure loss stops, or by removing the rail and injectors as an assembly to visually check for a dripping injector.