The match was a repeat of the London 2012 Olympic final, which Mexico won 2-1.
Mexico now have the opportunity to win bronze.
"We are saddened not to be in the final but all is not lost," Mexico midfielder Uriel Antuna said. "We now have to battle for the bronze medal. We want to take that medal home.
"We are a family and we fought until the end. Penalties are a lottery and we got the worst of it. We must bounce back and support those teammates that missed the penalties. They have all my support. The team is still united and we are going to fight for that medal."
Brazil, the holders having won gold in their home Olympics in 2016, had the better of the first half, but the second failed to live up to expectations.
The game went to extra time and, while Brazil looked more likely to break the deadlock, the match finished goalless after 120 minutes and it went to spot kicks.
Dani Alves scored Brazil's first penalty before Eduardo Aguirre saw his spot kick saved. Arsenal forward Gabriel Martinelli then put Brazil 2-0 ahead with a cool penalty and Mexico's second effort through Johan Vasquez hit the post.
"We had chances to win the game in regular time, but it wasn't possible," Brazil captain Dani Alves said. "I think fate wanted it to be that way. I have a lot of respect for the Mexico team, I like them a lot, but only one could go through and, luckily, it was us.
"Mexico is an opponent that we respect a lot because of the football school they have and how they fight in games. We have already suffered with this opponent. But this is football. We had a very good performance, we suffered when we had to, but I believe we deserved to be in this final.
"There's one game left. We need to maintain our focus and concentration. It's very difficult to win games when you face such different opponents. This requires a vast adaptation. We have a staff that controls that very well and gives us the necessary information and it has given results. We just need to take one more step."
Brazil coach Andre Jardine added: "We are to be congratulated for the game we played. We took no risks, we were solid. We looked for the goal all the time. The result, to go through on penalties, in my opinion, crowned the team that tried all the time to attack, to qualify in regular playing time.
"We deserved it, and in the end that was the feeling we got. If you deserve to win during the game, things happen on penalties. "Mexico have a great level. It's a team that defends very well and is also very dangerous in attack. It demanded a lot of concentration from us."
Brazil will face the winners of Tuesday's other semifinal between Spain and Japan in the gold medal match.