The electronegativity difference between $\text{N}$ and $\text{F}$ is greater than that between $\text{N}$ and $\text{H}$, yet the dipole moment of $\text{NH}_3$ (1.5 D) is larger than that of $\text{NF}_3$ (0.2 D). This is because:

In $\text{NH}_3$, nitrogen is more electronegative than hydrogen, so the individual $\text{N-H}$ bond dipoles point towards the nitrogen atom. The orbital (atomic) dipole due to the lone pair on nitrogen also points in the same direction, adding to the resultant bond dipole. In contrast, in $\text{NF}_3$, fluorine is more electronegative than nitrogen, so the $\text{N-F}$ bond dipoles point towards the fluorine atoms. The orbital dipole due to the lone pair points in the opposite direction, partially cancelling the resultant bond dipole and leading to a much lower net dipole moment for $\text{NF}_3$.