Am I holding my fret board wrong?

That seems like a really unnatural grip. Try relaxing your thumb and curling it around the curve of the back of the neck. That should be more natural and comfortable.