You must of missed this:
http://googlewebmastercentral.blogspot.com/2010/04/using-site-speed-in-web-search-ranking.html
"We encourage you to start looking at your site's speed (the tools above provide a great starting point) — not only to improve your ranking in search engines, but also to improve everyone's experience on the Internet."
Speed is a factor, right now it is not a major factor but it is a factor.