The Tigers led Vanarama National League South leaders Braintree at half-time on Saturday but ended up losing 3-1.
It means Gloucester are now five games without a win that leaves them in 15th place.
"It's been the story of our season so far that we struggle when things go against us," said Harris.
"We struggle to show character when we go behind. We know that we've got young lads and that we're in a very tough league with some very good sides but we need to grow up a little bit and take some responsibility.
"We know where we are and we know we lack the real quality and experience at this level. We've just got to keep battling away and stick at it. When we do well, we look a good side.
"We haven't got enough when things go wrong."
For more coverage, reaction and some of the best moments from games going throughout the season, head to our social media channels on Twitter, Facebook and now Instagram.