The books that made the biggest impact for me is a tough call. I suppose I'd have to say Ulrich Neisser's 1967 book Cognitive Psychology, whose focus and methods inspired my PhD and most of my career.
But the most memorable books for me are novels, and they do vary. On the shortest list I would have to place Peter Carey's Oscar and Lucinda for its sheer imaginative force, and Toni Morrison's Song of Solomon for its spectacularly beautiful writing.