Personally, I go back and forth on this issue - I think b/c the question itself is not that straightforward.
Does it matter? No, in the sense that it matters more "what you do," both in your time in school and afterward. Also, no in that graduate school might "matter" more. Not to mention that students will find good teachers almost anywhere.
However. Yes, in the sense that where you go to college can put you into contact with professors and probably even more significantly peers who will affect your own performance or "what you do" at school. Also, where you go to college and as a result, who you know, become important and meaningful forms of cultural and social capital.
Yes, in the sense also that, as one of the bloggers at Room for Debate also notes:
If you attend a highly selective college, the per pupil expenditure is $92,000, compared with just $12,000 at the least selective colleges. The richest colleges require students on average to pay just 20 percent of the total cost of college, compared with 78 percent at the least wealthy colleges.
I think another reason why I go back and forth on this issue is that I personally feel that I have gained, both materially and immaterially, from attending an elite institution - but that cannot justify the unfairness and inequity of the structure as it stands. Or as another blogger at Room for Debate succinctly states the problem:
Elite colleges are economically and personally productive for individuals lucky enough to attend them. The real issue is what this means for those who do not attend, and for the promise of upward mobility in our society as a whole.
It is not that the elite colleges don’t work. It is that they work too well as passive agents for the intergenerational reproduction of elites.
In these times, I think there is a serious problem of misapprehending the conditions that produce an individual's access and ability to attend an elite institution as that individual's merit. "Lucky enough" glosses over the so-called accident of birth intersecting with contrived particularities: I happened to be born in the United States as a result of my parents meeting during the medical residencies in New York City, following changes in American immigration law that themselves resulted from a "shortage" of doctors and nurses. I was raised in a so-called upper-middle-class suburb with "good" public schools where teachers and guidance counselors were familiar with the college application process, and took an interest in me - and so on.
When I think about what hopes I have for Beanie and Bubbie - I hope they will be as "lucky" as I have been. Which means not necessarily that I wish for them to attend elite institutions themselves, but that the promise of living lives of significance will remain in reach. For everyone.