Vacation time isn't mandated by law. In other words, employer do not have to give employees any paid vacation. Thus, only the contract for vacation pay, which is usualy in the form of a vacation policy controls the scenario. In this specific instance you ask whether paid vacation should count towards hours worked. The answer, absent reviewing the employer's policy which appears to be they do not credit it toward hours worked for the week, is that the wage laws only count hours physically worked toward the 40 hours per worweek. You can find this on the NJ DOL webpage here: https://lwd.state.nj.us/labor/wagehour/content/wage_and_hour_compliance_faqs.html#q5
As well you can find this in the Fed DOL regulations which state :
§785.7 Judicial construction.
The United States Supreme Court originally stated that employees subject to the act must be paid for all time spent in “physical or mental exertion (whether burdensome or not) controlled or required by the employer and pursued necessarily and primarily for the benefit of the employer and his business.” (Tennessee Coal, Iron & Railroad Co. v. Muscoda Local No. 123, 321 U. S. 590 (1944)) Subsequently, the Court ruled that there need be no exertion at all and that all hours are hours worked which the employee is required to give his employer, that “an employer, if he chooses, may hire a man to do nothing, or to do nothing but wait for something to happen. Refraining from other activity often is a factor of instant readiness to serve, and idleness plays a part in all employments in a stand-by capacity. Readiness to serve may be hired, quite as much as service itself, and time spent lying in wait for threats to the safety of the employer's property may be treated by the parties as a benefit to the employer.” (Armour & Co. v. Wantock, 323 U.S. 126 (1944); Skidmore v. Swift, 323 U.S. 134 (1944)) The workweek ordinarily includes “all the time during which an employee is necessarily required to be on the employer's premises, on duty or at a prescribed work place”. (Anderson v. Mt. Clemens Pottery Co., 328 U.S. 680 (1946)) The Portal-to-Portal Act did not change the rule except to provide an exception for preliminary and postliminary activities. See §785.34.
As you can see you must at least be on the premises or a prescribed place of work to be considered working. So it is clear you weren't working and it shouldn't be counted toward hours worked.
The second part, you ask whether the you should be credited back hours that exceed 40 for the week. The answer to that is totally dependent on the employer's policy; it's contract with you in regard to vacation pay. Nothing legally stops them from paying straight time for vacation hours in excess of 40 per workweek. So, only if their vacation policy prohibited the same would you be entitled to a credit back, in which case you'd have been overpaid and owe them back that amount.