Militao's first-half header put the visitors ahead after they had come under sustained early pressure from the hosts. Forward Dani Vega levelled with a superb solo strike after half-time, before late goals from substitutes Asensio and Isco saw Madrid through to the next round.
"Games like this are the ones that are most difficult," said Marcelo after the match. "We knew that these games can get complicated and we've learnt from it. We did a serious job today, everyone fought, we couldn't play a lot of football but we worked and fought.
"There are moments of games that are more complicated -- at the start, the first five minutes, and then early in the second half -- but we were able to control the game. Games like this are the hardest but they're the best ones to win."
Manager Carlo Ancelotti picked a strong team to face Alcoyano, who had caused an upset by knocking Madrid out of the cup at this stage last season.
Goalkeeper Thibaut Courtois was the only first team player to be rested, replaced by youngster Andriy Lunin. Karim Benzema, Luka Modric and Ferland Mendy missed out through injury while Vinicius Junior and Luka Jovic are yet to return after testing positive for COVID-19.
Madrid struggled in the game's early stages as Alcoyano created a series of chances. Goalkeeper Lunin tipped Vega's shot over the bar and Eduardo Camavinga blocked Francisco Miranda's goalbound effort.
Lunin turned Carlos Blanco's long-range shot around the post and Rodrygo fired across goal before Militao gave Madrid the lead in the 39th minute. The defender rose higher than Alcoyano goalkeeper Jose Juan to head Rodrygo's delivery into the net.
A quieter opening to the second half saw David Alaba shoot wide from outside the box, before Vega found the equaliser in the 66th minute, stepping inside Casemiro and Militao before curling left-footed past Lunin.
The home fans' joy was short-lived, though, as Asensio's shot deflected in on 77 minutes after the winger combined with Rodrygo and Eden Hazard.
Madrid's win was secured two minutes later when Isco -- brought on just moments earlier -- touched past Jose Juan, when Marcelo's cross had rebounded kindly to him off a defender.