Milan have now picked up 22 points after eight league games, their best record in the three-points-for-a-win era, going back to 1994-95.
"Verona were better than us in the first half, we were unlucky with the first goal but reacted after the break," Milan coach Stefano Pioli told Sky Italia.
"I saw the right expressions on the faces of the players at halftime, we had to step it up and we did. It was a great test of maturity."
There was further good news for the hosts when Zlatan Ibrahimovic, their top scorer last season, made his return from injury from the bench for the final 13 minutes after more than a month out.
"Everyone knows his presence is very important to us. He is not yet in top condition, I hope he can improve and play a lot of games," Pioli said.
They looked like a depleted force in the first half as stand-in keeper Ciprian Tatarusanu was forced into a save on an Ivan Ilic shot before Caprari smashed in the opening goal after seven minutes.
But the hosts were a team transformed in the second half and pulled one back when Giroud, making his first appearance since the end of September due to injury, headed in a Rafael Leao cross.
Kessie pulled Milan level with a 76th minute spot kick, and two minutes later Gunter had a moment he will want to forget quickly when a cross came in and he could not sort his feet out and accidentally guided a volley into his own net.
Ibrahimovic's return gave the home crowd another moment to cheer, and the Swede tried to make it a memorable comeback with a spectacular overhead kick attempt, but he did not connect properly.