I had the same problem with mine. At first, I just tried tightening the 2mm hex screw, but that didn't seem to make the mirror any less prone to being dislodged. After playing with it some more, I figured out how to get the mirror tight.
The trick as Richard alluded to is with the little plate on the back of the mirror, with 6 holes on it, that goes over the pivot ball on the mount. This entire plate actually screws into and out of the mirror body. The little 2mm allen screw is merely a set screw to keep the plate from backing out of the mirror once you get it adjusted.
Loosen that allen, then while holding that plate still (with your allen wrench stuck into one of the holes), rotate the mirror clockwise (from the glass side of the mirror) to tighten it. A quarter turn or so is probably all you need. Retighten the allen screw and then rotate the mirror to adjust it for proper rear view. It should be noticeable harder to move now. You may have to repeat this process to get it like you like it.