taken (one's) finger off of (one's) number

stopped watching something closely, resulting in a blunder. A Variation of lose one's number, with reference to a lottery number or some other number by or with which one may be identified, as an army number.