Keysher Fuller's 81st-minute effort, Costa Rica's first shot on target in a dreary game, stunned the drum-beating Japanese fans in the stadium and left the group wide open.
Japan's historic victory over Germany in their opener coupled with Costa Rica's 7-0 drubbing by Spain cleared the way for Japan to reach the last 16 with a game to spare.
But a much-changed line-up failed to spark and their hopes of reaching the knockout stage for a fourth time are now left hanging in the balance with their final game against Spain.
Costa Rica, written off before kickoff, are now level on three points with Japan and have a chance to reach the last 16.
Heavyweights Spain and Germany play later.
A first-half of mind-numbing tedium even tested the patience of the drum-banging Japanese fans massed in their blue shirts at one end of the Ahmad bin Ali Stadium.
Apart from a two-minute burst at the start of the game by Japan, the rest of the opening half was played at snail's pace.
For Costa Rica the lack of goalmouth action was probably most welcome seeing as they conceded three times in the opening half an hour against Spain.
Both goalkeepers were redundant and the only incidents that could pass as goalmouth action was a cross by Japan's Ritsu Doan that flashed across the goal and a scooped shot high over the crossbar by Costa Rica forward Joel Campbell.
Japan's coach Hajime Moriyasu took off defender Yuto Nagatomo at half time and replaced him with attacker Takuma Asano who was surprisingly left on the bench despite scoring the winner against Germany.
Japan were camped in Costa Rica's half but and it seemed only a matter of time before their pressure would pay off but in a rare break, Japan failed to clear and Fuller curled a shot which keeper Shuichi Gonda could not keep out.