One argument is that refactoring can be an alternative to upfront design. In this scenario you don't do any design at all. You just code the first approach that comes into your head, get it working, and then refactor it into shape. Actually, this approach can work. I've seen people do this and come out with a very well-designed piece of software. Those who support Extreme Programming often are portrayed as advocating this approach.
Although doing only refactoring does work, it is not the most efficient way to work. Even the extreme programmers do same design first.