Neymar netted his 77th international goal in extra time -- moving him level with legend Pele for the country's all-time men's goal record -- but it proved not to be enough as Brazil suffered a second-consecutive quarterfinal exit at a World Cup finals.
Tite's side were frustrated for much of the match by a dominant display from Croatia's midfield trio -- Marcelo Brozovic, Mateo Kovacic and Luka Modric -- and some inspired form from goalkeeper Dominik Livakovic during a cagey affair that was sent to extra time when the deadlock could not be broken in regulation time.
Neymar, who had been quiet until then, thought he had struck an extra-time winner when he skipped past Livakovic and powerfully fired into the roof of the net, but 2018 runners-up Croatia hit back with just four minutes to play as Bruno Petkovic levelled the score at 1-1 with a deflected strike from the edge of the box.
Brazil came unstuck in the shootout, with Rodrygo missing their opening penalty and Marquinhos missing their fourth, while Croatia converted all of theirs to claim a memorable win.
"This is just for the Croatian people," coach Zlatko Dalic said. "A great match from the first minute to the last. We eliminated the biggest favourites. This is not the end for us, let's keep going.
"This is a victory for all of us, everyone gave their best. Thanks also to those on the bench, who also live for this national team."
Croatia initially refused to be intimidated by their more illustrious opponent and in the first half successfully stifled most Brazil moves. Apart from a weak Vinicius Jr. shot, Brazil had little else to offer in terms of attack with Neymar off the pace and shut out by a hard-working backline.
It was Croatia who looked sharper and more skilled in their passing game, and by the end of the first half they had the upper hand in terms of possession.
A Neymar free kick just before the break did not trouble Livakovic as Brazil, whose last win over a European opponent in the knockout phase of the World Cup came in their 2002 final victory over Germany, struggled to find space.
They did, however, almost take the lead three minutes after the restart when Croatia midfielder Josko Gvardiol tried to clear the ball but almost turned it into his own goal.
Seconds later, claims for a hand ball by Croatia defender Josip Juranovic were shot down following a VAR review before Neymar was sent through by Richarlison in the 55th minute only to have his close-range effort blocked by Livakovic.
Brazil enjoyed a much better start to the second half and came close again in the 66th when Lucas Paqueta was denied by Livakovic.
The Croatia goalkeeper was kept busy and stopped Neymar in the 76th minute and by that time had more saves in the match (seven) than his Brazil counterpart, Alisson, had in the entire tournament (five).
Livakovic parried another Paqueta shot five minutes later as the few hundred Brazil fans behind the goal grew louder and more impatient.
The game trickled to extra time with Brazil coach Tite appearing to be visibly nervous, but Neymar's strike had them dreaming of the semifinals.
Information from Reuters contributed to this report.