Brundage was an all-around athlete, competing in the 1912 Summer Olympics in Stockholm in the pentathlon and decathlon events, finishing 6th and 16th, respectively. He also won the US national all-around title in 1914, 1916 and 1918.
In 1928, Brundage became president of the Amateur Athletic Union (AAU). He became the president of the United States Olympic Committee (USOC) in 1929 and gained the vice-presidency of the International Amateur Athletic Federation (IAAF) in 1930.
As USOC president, Brundage rejected any proposals to boycott the 1936 Summer Olympics in Berlin, where German Jews were excluded, and became a member of the International Olympic Committee after the group expelled American Ernest Lee Jahnke, who had urged athletes to boycott the Berlin games.
On the morning of the 400-meter relay race, at the last moment, the only two Jews on the 1936 US track team, Marty Glickman and Sam Stoller, were replaced by Jesse Owens and Ralph Metcalfe. Glickman later said that that decision might have been the result of pressure from Brundage.
Brundage later praised the Nazi regime at a Madison Square rally, and was expelled from the America First Committee in 1941 because of his pro-German leanings. After the 1936 Olympics, Brundage's company, the Avery Brundage Company, was awarded a building contract by Nazi Germany in 1938 to build the German Embassy in America. Brundage was notified in a letter from Nazi authorities acknowledging Brundage's pro-Nazi sympathies. As late as 1971, after many revelations over Nazi Germany's use of the 1936 Olympics for their own propaganda, Brundage still claimed "The Berlin Games were the finest in modern history...I will accept no dispute over that fact".
During his tenure as IOC president, Brundage strongly opposed any form of professionalism in the Olympic Games. Gradually, this opinion became less accepted by the sports world and other IOC members. It led to some embarrassing incidents, such as the exclusion of Austrian skier Karl Schranz, who was accused of being a professional, from the 1972 Winter Olympics.
He opposed the restoration of Olympic medals to Native American athlete Jim Thorpe, who had been stripped of them when it was found that he had played professional baseball before taking part in the 1912 Olympic games (where he had beaten Brundage in the pentathlon and decathlon). Despite this, Brundage accepted the "shamateurism" from Eastern bloc countries, in which team members were nominally students, soldiers, or civilians working in a non-sports profession, but in reality were paid by their states to train on a full-time basis. Brundage claimed it was "their way of life."
It was revealed after his death that Brundage had been responsible for notifying the IOC of Thorpe's playing professional baseball years before.Brundage opposed the inclusion of women as Olympic competitors; he insisted they have no role in the Olympic Games beyond the ceremonial or decorative. He was quoted in 1936: "I am fed up to the ears with women as track and field competitors... her charms sink to something less than zero. As swimmers and divers, girls are [as] beautiful and adroit as they are ineffective and unpleasing on the track." (Quoted from The Ultimate Book of Sports Lists, by Andrew Postman and Larry Stone, 1990
Brundage also opposed anything that he viewed as the politicisation of sport. At the 1968 Summer Olympics in Mexico City, US sprinters Tommie Smith and John Carlos raised their fists to show support for the Black Power movement during their medal ceremony. Brundage expelled both men from the Olympic Village, suspended them from the US Olympic team and had them stripped of their medals. It must be noted that Brundage oversaw the IOC during the most eggregious violations of amateur sportsmenship during the rise of state funded athletics in the eastern bloc.
He may be best remembered for his decision during the 1972 Summer Olympics in Munich, Germany, to continue the Games following the 5 September Palestinian terrorist attack which killed 11 Israeli athletes. While some (such as Israel) criticized Brundage's decision, most did not, and few athletes withdrew from the Games. The Olympic competition was suspended on September 5 for one complete day. The next day, a memorial service of 80,000 spectators and 3,000 athletes was held in the Olympic Stadium. Brundage gave an address in which he stated
"Every civilized person recoils in horror at the barbarous criminal intrusion of terrorists into peaceful Olympic precincts. We mourn our Israeli friends [...] victims of this brutal assault. The Olympic flag and the flags of all the world fly at half mast. Sadly, in this imperfect world, the greater and the more important the Olympic Games become, the more they are open to commercial, political, and now criminal pressure. The Games of the XXth Olympiad have been subject to two savage attacks. We lost the Rhodesian battle against naked political blackmail. I am sure that the public will agree that we cannot allow a handful of terrorists to destroy this nucleus of international cooperation and goodwill we have in the Olympic movement. The Games must go on...."
– Simon Reeve, "One Day in September" (2000)
To this day the IOC has never officially recognised this tragedy during any of the games held since.
Brundage strongly opposed the exclusion of Rhodesia from the Olympics due to its racial policies: after the attacks in Munich, Brundage linked the massacre of the Israeli athletes and the barring of the Rhodesian team (see above). He later apologized for the comparison. Brundage is also remembered for his proposal of elimination of ALL team sports from the Summer Olympics. He also wanted to eliminate the Winter Olympics entirely.