The defeat -- Barca's third of the season -- comes just days after they had been beaten 2-1 against Real Madrid in El Clasico on Sunday.
Barca dominated possession but struggled to mount attacking pressure for much of the match -- the side's first shot on target came on 71 minutes when Memphis Depay stepped up to take a penalty, but Rayo goalkeeper Stole Dimitrievski saved the attempt.
"The game that we lost today is not down to attitude, not how we played because no other team created as many chances against them. So it's a question of not scoring. I can't say anything else," Koeman said after the match, hours before his firing.
Barca had a flurry of late chances to get back into the game. The first came in the final two minutes of stoppage time as striker Sergio Aguero headed wide from a Sergino Dest cross. Teenage midfielder Gavi, who had come off the bench to replace Philippe Coutinho, then mishit a volley from just inside the box moments later as Rayo secured the first win over Barca since 2002.
"I'm really emotional right now. It's my first win against Barcelona after playing four times against them [with Atletico]," a tearful Falcao told reporters.
"Seeing the fans singing, the joy that they are having in the stands, it's humbling and satisfying."
The Catalan side sit in ninth place in LaLiga, and next face Alaves at Camp Nou on Saturday.