More than 645,000 people never received their stimulus payments, according to the Treasury Inspector General for Tax Administration.