Do you not understand the concept of limit? No one is saying you switch from finite numbers to an infinite number while counting, but the limit of counting numbers is infinite.
I'm saying that your usage of an infinite set to describe the situation is wrong. It has the count going 1+1+1+1...N. Doing that goes on forever but never gets to the end of finite numbers or arrives at infinity. If you represent that with a set which contains infinity, your set does not accurately represent what you are attempting to represent with it.
If you take 3+3+3 and represent it as 3*3 or 32, you get an answer of 9 any way that you do it. If you use some mathematical concept to represent 3+3+3 and the result of using that concept arrives at something other than 9, your concept does not accurately represent 3+3+3. If you do 3*3 and get 10, you're not exposing some kind of paradox inherent in the multiplication tables, you're just doing multiplication wrong. Any way that you correctly represent the equation comes out to the same answer.
It's the same here. If you keep adding one and halving the time, you never arrive at either infinity or noon. If you use a set as a shorthand way to represent that and you arrive at a different answer than if you just do it the long way with sequential addition, your set is wrong and doesn't contain the equation you tried to put into it anymore than a set which was supposed to contain 3+3+3 but ends up equalling 10 isn't actually a set with 3+3+3 in it.