On the 125th anniversary of the club's foundation, Newport were at risk of being knocked out of the competition when Brighouse took the lead in the 22nd minute, after a defensive error.
They were unable to really test the Yorkshire side but in a much-improved second half performance the breakthrough finally came when a cross from the goal-line found the head of Tom McInnes, who guided home the equaliser.
And a minute later the 446-strong crowd were in raptures, when McInnes toe-poked the ball into the net from Aleks Przespolewski's cross to take the lead.
There were nervous moments and an agonising five minutes of injury time for Newport to ride out but they held on well, triggering huge celebrations.
Manager Steve Brougham said afterwards: "We talked at half time and we knew we hadn't done ourselves justice in the first half. We knew if we carried on we would go out with a whimper, so we stepped up and it's just brilliant. The changing room right now is an amazing place to be and we cannot believe we are in the last 16."
Prior to the game Newport were presented with a signed Liverpool shirt and a signed picture of Liverpool boss Brendan Rodgers - a former Newport player - proclaiming "Up the Islanders!".
Newport face either Borrowash Victoria or Ascot Town in the fifth round at St George's Park on February 9.
In Division 1 of the Sydenhams Wessex League, there were two 4-0 wins to celebrate.
Leaders Cowes Sports defeated Laverstock and Ford, with goals from Dom Burbidge, two, Tom Dye and Gareth Bricknell.
East Cowes Vics beat Ringwood Town with goals from Simon Williams, two, Tom Scovell and Sam Yelland.