The O’s lead 3-0 at Dover with just under half an hour to go thanks to goals from George Elokobi, Macauley Bonne and Matt Harrold.
The Whites fought back to level, however, through a Ryan Bird double and Mitch Pinnock goal.
But Edinburgh’s men grabbed the winner in the last minute thanks to David Mooney - a goal that sent the O’s into the last eight.
The boss was far from happy that his side let their sizeable lead slip and says they were their own worst enemy at times.
“I think we were a couple of minutes away from the best performance since I’ve been here,” he told the club’s website.
“We’ve won the game with what was quite an incredible finish. The result was nothing we didn’t deserve but we were the instigators of our own problems.
“There were a lot of positive but also some real poor moments in the game that frustrated me. I don’t want to see them creep back in because of the strides and improvements we’ve made.
“There’s lots to reflect on but we’re obviously delighted to be into the next round.”
For more news, reaction and action from across the National League, like us on Facebook and follow us on Twitter and Instagram.