Check what FPSs do you have ingame by using any 3rd party program (fraps or whatever).
I had some weird problems on my:
Intel core duo e8400 (3,0 ghz)
2 GB ram
where with NONE changes in my settings, FPS dropped down to 20 or 15... sometimes even to 1 frame per 5 seconds... after playing around in settings, I changed something from Hight to Medium then back to High, I received boost to 200 fps (turned off limit in config file).
First time, game looked pretty smooth, but mouse didn't (~25 fps). Right now, with 90-120 FPS I can finally play it how I should do it in first place
Play around SSAO, Parallax and Shadow resolutions (turn them off and turn them back on...mostly works with Parallax and shadow resolution set to medium and back to high :O ). Maybe they lag your controls somehow.
Check for vsync if it's being turned off (force it off in nvidia's control panel).
Unlock more FPS in config ini located in "Documents" folder of your system... maybe this will help.