The former Hull City and Bolton Wanderers manager has departed following the 2-1 play-off semi-final defeat to Chester.
Harriers, who missed out on the National League North title on the final day of the season, will now begin the search for his replacement.
Brown’s assistant Neil McDonald also leaves, a year after they both signed new two-year contracts.
In a statement shared on their website, club Chairman Richard Lane said: “Whilst we came very close to achieving our goal, the outcome of the season ultimately proved bitterly disappointing.
“We remain firmly focused on our primary objective for the club, which is returning to the National League, we feel now is the right time to make changes before embarking our future campaign.
“I would personally like to thank both Phil and Neil for their efforts during their time with Kidderminster Harriers and we wish them nothing but the best for the future and their next challenge.
“The process of appointing a new manager is already underway and we hope to update our loyal supporters on that soon.”