And Australia's hopes at the tournament are also over after Canadian teenager Eugenie Bouchard beat local hope Casey Dellacqua in straight-sets on Sunday night.
With Williams struggling with a back injury, the Serbian former world No. 1 Ivanovic pulled off what is likely the biggest individual win of her career that includes a French Open title when she won in three sets on Rod Laver Arena.
World No. 4 Li Na and 28th-seeded Italian Flavia Pennetta are the other women through to the quarter finals so far on Sunday.
The women's fourth round action began on Rod Laver Arena on Sunday with 31-year-old Pennetta taking on German ninth-seed Angelique Kerber.
The match went to a third set, but it was the resurgent Italian who prevailed 6-1, 4-6, 7-5 victory to advance to the Australian Open quarter finals for the first time in her career.
It was a tremendous performance from Pennetta who hit eight aces and 43 winners while breaking her opponents serve on eight occasions.
The loss for Kerber certainly wasn’t helped by 35 unforced errors, four double faults and winning just 5-of-13 points at the net,
The victory continues Pennetta's remarkable career resurgence that included her first ever semi-final appearance in a Grand Slam at last year's US Open.
Up next on Rod Laver Arena was a battle between world No. 1 Williams and former world No. 1 Ivanovic.
Williams had cruised into the fourth round in Melbourne without dropping a set and she had not lost a set previously in her four wins over Ivanovic either, but that all changed on Sunday.
Ivanovic lost the first set to Williams, but then played perhaps her best ever tennis since winning the 2008 French Open as she recorded the 4-6, 6-3, 6-3 victory.
Ivanovic managed to win more games in the match (14) than Williams' previous three Australian Open opponents had managed put together.
The 26-year-old 14th-seed is now into the quarter finals at the Australian Open in what is only her second time doing so at a Grand Slam in her 21 majors since claiming the title at Roland Garros in 2008.
There was also a women's singles Round 4 clash on Hisense Arena that saw a thrashing with Chinese world No. 4 Li Na thumping Russian 22nd-seed Ekaterina Makarova 6-2, 6-0.
It was a crushing win for the 31-year-old Chinese star who has enjoyed a great run of success in Melbourne including reach two finals.
She will now face fellow veteran Pennetta in the quarter finals.
The final women's fourth round match on Sunday was on Rod Laver Arena with Australia's world No. 120 Dellacqua taking on Canadian 19-year-old Bouchard.
The first set went to a tie-break where the Australian managed to prevail seven points to five, but from there the 30th-seeded teenager took charge of the match to show that she could very well be a title contender despite competing in her very first Australian Open.
Both women exchanged an early break of serve in the second set to have things tied up at 2-2, but from there Dellacqua's game came apart.
However, even more so was the fact that Bouchard played at a level that very few women could compete with in the game at the moment as she won the last 10 games straight of the match to come out victorious, 6-7 (5), 6-2, 6-0.
It was a commanding performance from the young Canadian who has only ever competed at three Grand Slam tournaments before.
It was a commanding performance from her against Dellacqua as she sent down six aces, landed 65 per cent of her first serves winning 77 per cent of them, and hitting 29 winners, claiming 10-of-12 points at the net and breaking the Australians serve six times.
Bouchard is now through to her first Grand Slam quarter-final where she will take on Williams' conqueror, Ivanovic with a berth in the semi finals up for grab.
Despite the loss, the fourth round appearance equalled Dellacqua's best ever Grand Slam performance as the 28-year-old left-hander now looks to keep the momentum going through the rest of 2014.