Refactoring Kata Tennis to State Pattern

Last month I posted about the Tennis Kata and the last Katayuno organized by Softonic.
In each of the four iterations, we used TDD and pair programming to develop the exercise from scratch.
We came to understand the tennis game as a state machine, (see the diagram showed below), and created tests for all the transitions.

We didn't have time to finish the exercise, but, once at home, I redid and finished it.
I commited the result to a public Bitbucket repository stating in the initial commit message that I'd like to refactor the code to the State Pattern to see how far the state machine idea could go.
Before refactoring to the pattern, I had to remove some duplication, rename some method and variables and introduce a Player class. Finally this morning, I was able to do it. It was very nice to observe how all the pieces started to fit together and how the code got simpler.
Thinking about the process retrospectively, I'm under the impression that the transition from the TDD resulting code to the version with the state pattern was not very difficult because the idea of the game as a state machine was there all the time. I wonder how much more difficult would have been to refactor to the state pattern, if the code of the initial solution hadn't followed the machine state idea.
This makes me reflect on how TDD is done and how your view about the problem and your knowledge and experience in refactoring and design can make completely different solutions emerge. I think that some of these solutions, even though they pass all the tests, can paint yourself in a corner from where you will need epic refactoring sessions to get out.
I think that the refactoring step is crucial for TDD success. We should not forget that in TDD we're designing not making tests pass, so we need to make a bit of "small design upfront" in each TDD cycle when creating new tests and when refactoring.
Design does not emerge on its own, we make it emerge. To do that we need to have some intuition or idea about where we'd like to go with the next TDD cycle.
I heard Jason Gorman say once that "Refactoring is the fairy dust that makes TDD magic work". The more I practice, the more I think he is right.