It's no different than blood cell count.
Your body makes sperm at X rate. If you have too much backed up, you might have a "wet dream" to dispose of excess. After you have an ejaculation (or several), you are expelling sperm at Y rate. If you deplete faster than you create, your sperm count will be lower.
There's nothing you need to do about this. If you are going to have an official sperm count performed, I'm sure they tell you how long prior to not have sexual/masturbatory activity to ensure you didn't throw the test off.
If you follow their guidelines and your sperm count is consistently low, then you see about what can be done to raise sperm production.