I'm getting huge hits to the swapbuffers while running through my map, I've checked the wiki but can't really figure it out (fill-rates:confused: )
Does anyone know how to fix this?

I believe it has something to do with you pushing the abilities of your vid card to the max.
I was having the same problem on a map and searched for a really long time and couldnt find much. There is a thread on the steam forums that has got some info. I'd do a search for you, but it's early and I need to get to work!

Can be caused by a lot of things. Too many reflective surfaces, bump mapping, smoke, fire and other effects. But also too many different textures/props in one place, meaning they don't fit into your gpu's RAM, so it has to swap quite a lot with the main RAM.

Try those commands:
mat_bumpmap 0
mat_specular 0
They switch off reflections and bump mapping.

If swap buffers stay high, try a compile without water, windows and any particle effects.
If that doesn't do it either, you'll probably have to optimize your map some more.

It seems to be the specular, what does specular do? I have a lot of snow in the map, is it related?

Uses environment mapping too ya. Could be the reason, especially if you use way more than the stock maps. Maybe try a different kind of snow texture, one without that envmap stuff.