There are many reason not to.
First, it breaks your listener concentration. It’s a small thing at first, then your listener start noticing your err instead of content of your talk. For every err they heard, their annoyed level increased. They are getting not focused and then they started counting on how many times you said err in your talk. Instead, keep it silent, put your serious pose, that made your listener increase their attention, or give time for them to absorb what you have said.
Second, it make you sound less serious. Have you prepared it? Do you know what you are talking about? It also looks silly, and unprofessional. Prepare for every talk you are going to do, even if it’s only five minutes talk. Think it as investment, that you will be giving speech to thousands of people, and this is a preparation for that. You will be giving speech, it might be on your birthday event, or your wedding, or your company speech, or your child graduation. You might hate it, but it is important to others that you talk.
Third, it make you harder to think and concentrate. When you start using too much err in your words, it might mean you are getting nervous, your blood rising up, you might be sweating. If you are realizing it, take a deep breath, start talking slowly, one word at a time. Build your pace to comfortable level.
Fourth, it make you ignorant to your listener. You will focus on sound of you talking with err, instead of engaging your listener, either with your body language, or your word intonation.
Giving speech is a skill that can be trained and mastered. You might have this habit, but you can learn to stop it, or decrease it.
But you know what? I’m in no way have mastery of giving speeches, and what I wrote in this post are bull shit, random thing that have no scientific bases, and a pinch of my experience. Well, shit still can be used for fertilizer, so it is far better than trash. May my shit fertilize your way in giving talk. Happy learning!