You have to send packets to the server and the server has to send packets to you so you can never get rid of lag entirely.
Since everything is going through optic fiber cables these days that means the fundamental hard limit is the speed of light. Google says light needs 40 milliseconds from Paris to New York so that would be a theoretical minimum amount of delay between these two places, but you'll never get there in practice.
There are ways to make it appear lower, for example the game tries to predict where other players are going (that's why you sometimes see players teleporting around if lag gets bad).
Now if a lag spike appears that means some additional delay happened along the way. Maybe your router is congested or maybe your internet service provider is, or maybe they had a hardware outage. It can be a lot of things. More often than not it's just game servers being buggy though.
Obviously in real-time applications like games you are going to notice disturbances a lot more than say, when browsing.