When you put it that way yeah... withdrawal of consent should be explicit, but doesn't that mean that implied consent shouldn't be a thing either?
It's a dangerously grey area though because as I mentioned earlier, people implicitly consent to sex all the time, heck, I doubt that married couples have to make verbal agreements every time they initiate sexual contact. I dunno man.. we've come a long way from "it's not rape if you're married" and there has to be a good middle ground somewhere...