Simon, are you positioning the LAA so it is inserted all the way down between the jaws so the adapter is physically contacting and resting on the top horizontal surface of the jaws. Also, the vertical sides of my LAA are close or in contact with the vertical sides of the jaw set, too.
Positioned like this I have no wobble or movement. The LAA is clamped tightly, stable and securely. Have you tried to increase the tension adjuster’s setting so it squeezes the LAA more tightly as it’s clamped tightly.
I usually use a small rectangular patch I cut from a real leather Chamois cloth to augment the hold. It is placed between the vise jaws and what ever I’m clamping. The dry chamois as it compresses in the clamping jaws improves the hold on whatever is clamped.