When owners or board members are unhappy with a particular board member (or members), there is generally a method by which they may be removed. The association's Bylaws will provide specific information about the process.
Some Bylaws allow for removal by a vote of the board members. Others allow for removal by a vote of the owners. Refer to your association's Bylaws to determine how a board member may be removed.
Remember, all associations must have a minimum of 3 board members (your Bylaws may require more). So any board members who are removed will need to be replaced in order to remain in line with your governing documents.