Brown will now focus on his position as assistant coach of the San Antonio Spurs in the American NBA.
The decision to step away from the international game will also allow him to spend more time with his family in the United States.
"I inherited a strong Boomer culture moulded by many committed, outstanding coaches and players that have come before me," said Brown in a statement.
"My staff and I are confident that the current program is well placed to again be brought forward."
The coaches of Australia's national basketball teams are contracted for four-year terms ahead of every Olympic Games.
Basketball Australia will begin the search for a new men's coach once they complete a review of the program leading up the recent London Games.
Brown said it had been a privilege and a pleasure coaching the Boomers.
"I am especially proud of the commitment that Australia's best players made to the Boomer program during this term," he said.
"I leave with fond memories of a group of players who played with a spirit and toughness that earned the respect of the global basketball fraternity.
"I am proud to have been their coach.
"The basketball community should be excited to have our current group of core players and new talent emerging throughout the country."
BA chief executive Kristina Keneally said Brown would leave a lasting legacy.
"Brett's contribution to the men's national team program has been profound - it will last long after his departure," Keneally said.
"Under Brett's charge we've seen the Boomers surge forward in leaps and bounds - highlighted by the emergence of the likes of Joe Ingles, Matthew Dellavedova and Aron Baynes as star players in our run at the 2016 Rio Olympic Games."
Before joining San Antonio, Brown coached for nine years in the NBL, winning the title with the North Melbourne Giants in 1994.
He spent eight years as an Australian assistant from 1995-2003 and replaced Brian Goorjian as head coach after the 2008 Beijing Olympics.