I once had problems installing a GTX570M into an old M15x. Eventually I worked it out. Did you get the file from laptopvideo2go? That's where I eventually got it sorted.
There is another option though which can be easier. I once bought some GTX780M's on ebay for my M18x R2. The store was upgradeyourlaptop. They have all the modded drivers for different cards for different Alienwares. Hopefully
this link will lead you in the right direction. Click on the drivers link and you will find one for your M15x and the 660M. Good Luck!