Neymar was among five players sent off after a fight broke out at the end of the match and said on Twitter that he was racially abused by Gonzalez.
Le Graet claimed that racism in French football was non-existent and said: "When a black player scores a goal, the whole stadium is on its feet."
Evra responded to Le Graet's comments and also claimed that himself and fellow teammates experienced racism while on international duty with Les Bleus.
"We have to get Le Graet fired, because he really messed up," he said in an Instagram video.
"I am obliged to reply to Noel Le Graet, I'm obliged to speak of the Chateau [France training base Clairefontaine], you know very well what happens there, how many racist letters do we receive?
"One said 'Didi, take your monkeys and go to Africa.' How many messages like that do we receive? Yet, we hide them, I have seen some.
"We have even received boxes full of faeces. We have assigned places to eat, but every time the president or the politicians came, everything changed.
"I was sitting there and all of a sudden they would put me at the end of the table, where normally [Mamadou] Sakho and [Bacary] Sagna were, we had to change.
"They put Hugo Lloris, Laurent Koscielny and the president in the middle, but we knew they were the rules of the game, when the president wanted a photo it was better to be with Lloris and Koscielny instead of Sakho or Sagna."
In January, Evra revealed that he had received a letter of apology from Liverpool after he was subjected to racial abuse from Luis Suarez in 2011 which saw the forward subsequently banned for eight matches.