Battery Spencer in the Marin Headlands is unequivocally THE coldest spot in San Francisco. "No no no, Twin Peaks is, for sure!" you might say...but you'd be wrong. I've been up on this spot with a thermal undershirt, t shirt, sweater, and thick jacket on a 70 degree day (I always tend to bring the heat with me to the bay area when I visit) and was still freezing.
That said, bearing with the cold blistering winds is always worth it for what I think is the absolute best view of the Golden Gate Bridge. You can get a little closer via the path you can see on the left, but at this higher vantage point you get to experience the bridge in it's full majestic glory. Not to mention, if an errant wind gust decides to send you flying over, at least up here you have some land to crash into before any steep descent sends you plummeting to your doom!