This is completely possible to perform without upgrading your gun despite the achievement icon being the upgraded Gauss Gun.
Grigori teleports to the edge of the screen whenever he is hit 3 times.
He starts on the right side of the screen and so long as you hit him quickly regardless of which side he is on you should not need to teleport.
You will need to jump slightly in order to hit his fighting pod with your shots.
Grigori only has 1 attack which you would need to teleport to avoid which he will never attempt if he is beaten quickly.