Two books I like to have close by are Martin Fowler's "Refactoring: Improving the Design of Existing Code" and Joshua Kerievsky's "Refactoring to Patterns". It's not that productive to memorize all of the refactorings, and especially the patterns. Applying patterns and refactorings before they are needed can make your code unnecessarily complex and less maintainable in the long run. No one likes running into an Abstract Factory with only one concrete factory implementation.
I need a context for which to apply patterns and refactorings, and code smells provide the best indication for those contexts. All I really need to do is get good at recognizing smells, and then it becomes a cinch to look up possible refactorings. I made a nice quick reference guide help me match smells to refactorings:
It's adapted from this guide, but I had a tough time reading that one when it was pasted on the wall next to me. My version spans 3 pages in a larger font and a landscape layout, and I've had no issues reading it on the wall next to me.
The list of smells is much smaller and easier to remember than the patterns and the refactorings. By recognizing smells instead of using patterns when they're not necessary, I can apply an evolutionary, JIT (just-in-time) approach to my design and architecture. Instead of planning patterns beforehand, I can have confidence that I can recognize where they're needed, and apply them as necessary, exactly when they're necessary, at the last responsible moment.