So here's where things go bad. If you run VirtualBox and you run a Windows 10 system, you have to turn this off. I believe I had this on due to an early version of VirtualBox not handling ctrl+left click, but they seem to have fixed this in the latest version... ironically my workaround that fixed my issue now breaks it!