Ranked 48th, Mattek-Sands upset the world No.5 in straight sets, 7-5 6-3, in 92 minutes to leave Radwanska's Australian Open preparations in tatters.
Her defeat marked the first time the women's titleholder has lost her opening match in Sydney since 1990.
The former Wimbledon runner-up hasn't won a set - let alone a match - since October after losing all three of her round-robin encounters at the season-ending championships in straight sets.
Radwanska's worrying form slump is a far cry from last year when the Pole arrived at Melbourne Park for the season's first grand slam riding a nine-match winning streak after picking up back-to-back titles in Auckland and Sydney.
Li Na eventually stopped her streak at 13 with victory over Radwanska in the Australian Open quarter-finals.
Mattek-Sands' surprise second-round win on Tuesday - after Radwanska enjoyed a first-round bye - thrust her into a quarter-final with either fellow American Madison Keys or Croatian wildcard Ajla Tomljanovic.
Mattek-Sands needed four set points to clinch the opening set but rolled through the second in barely half an hour with three service breaks.
German fifth seed Angelique Kerber is also through to the quarter-finals after downing big-hitting Estonian Kaia Kanepi 6-3 6-4 in her second-round match.
Danish sixth seed Caroline Wozniacki plays later on Tuesday against Czech Lucie Safarova.