Spurs missed the chance to leapfrog Chelsea into the top four as Spurs move into fifth place and Newcastle remain in 17th place, just three points above the bottom three.
"Whenever you're ahead in a game, as we have been a lot this season, and you give the lead away in the last five to 10 minutes, it's always hard to take," Kane told BBC Sport.
He added: "I feel like we could have played better in the second half. We were a bit sloppy. We didn't take our chances and got punished for it."
Spurs responded and thought they opened the scoring but Carlos Vinicius' strike was ruled out for offside.
The hosts did find the breakthrough after 28 minutes after Sean Longstaff capitilised on some poor Spurs defending to set up Joelinton to score.
Less than a minute later, it was Spurs who benefitted from Newcastle's defensive mistake as Kane walked the ball into the net after a mix up inside the box.
Kane put Spurs ahead four minutes later when he latched onto Tanguy Ndombele's pass and struck the ball into the bottom corner.
Newcastle came close to an equaliser after the break but Lloris produced a fingertip save to stop Miguel Almiron's curling strike.
Both sides had good chances to score after the hour mark but Japhet Tanganga's header was cleared off the line and Joelinton's close-range header went wide.
With five minutes remaining, Kane came close to clinching his hat trick but his strike came off the post.
Moments later, Newcastle grabbed an equaliser as Willock reacted quickest in the box to dent Spurs' chances of finishing into the top four.
"I'm an Arsenal player so it does makes it a bit sweeter," Willock, who came on in the 79th minute, said. "[Manager Steve Bruce] told me to come on, run around and get in the box to try and score goals. I'm happy I was able to do that."