From TopEndSports.com: 

It is fascinating the number and variety of sports that have at one time been part of the Olympic Games. Since the first modern Games in 1896, 12 sports have disappeared completely from the Olympic schedule. These are golf, rugby, croquet, cricket, Jeu de Paume, pelota, polo, roque, rackets, tug-of-war, lacrosse and motor boating. Added to this is a large number of discontinued events from sports that are still on the program, such as the swimming events of plunge for distance and underwater swimming, the track and field events of the standing highjump and tug-of-war, and the shooting events of pigeon shooting and duelling pistol.

You can have a look at the complete list of all discontinued sports, or my list of some the more unusual sports.

In the past, the host country often included their own sports. Some of the discontinued events were simply updated to metric distances or modified into the well-known events of today, especially in track and field and swimming. There are also some interesting sports that were included as demonstration sports, some eventually becoming part of the official program, others disappearing as they should. See more about demonstration sports and the complete list of demo sports.