Since a finite group is nilpotent if and only if it is the direct product of its Sylow subgroups, it suffices to show that every Sylow subgroup of G is normal.

Let P be a Sylow p-subgroup of G. Since [G: N_G(P)] < 2, it follows that N_G(P) is normal in G.

Since P is the only Sylow p-subgroup of N_G(P), it follws that P is a characteristic subgroup of N_G(P). So since P is a characteristic subgroup of a normal subgroup of G, it follows that P is normal in G.