So here's where things go bad. If you run VirtualBox and you run a Windows 10 system, you have to turn this off. I believe I had this on due to an early version of VirtualBox not handling ctrl+left click, but they seem to have fixed this in the latest version... ironically my workaround that fixed my issue now breaks it!
Thursday, December 3, 2015
Karabiner allows you to disable ctrl+left click on OS X. That will, by default, do the same thing that right-click does with a mouse. You choose the following option to disable it: