"You're playing games with the public. Everyone has known this since you got elected," the unnamed man said in a BBC video.
The video, posted on Friday, showed Johnson -- one of Brexit's chief supporters -- attempting to defend his handling of the issue.
"What I think the people want us to do is to leave the European Union," Johnson told the man. The man claimed that everyone knew that ,but that Johnson wasn't engaging in negotiations like he should.
"You should be in Brussels negotiating," he said, referring to the Belgian capital that also serves as the de facto capital for the European Union.
He complained that rather than "leading the British people" in Brussels, he was in Morley. "You should be in Brussels, leading the British people to negotiate against leaving the European Union."
"He's been speaking for the last six weeks since he's been elected and he's said nothing," the man said later in the video. He added that Johnson had "gone into Parliament" and "played games."
Johnson replaced Theresa May as prime minister in July but faced pushback over his call for Britain to leave the E.U. with or without a deal on Oct. 31. On Friday, the House of Lords passed a bill conditioning Britain's departure on reaching an agreement with the E.U.
Johnson told the man in Morley that he was "on the verge of getting a deal." When the prime minister called on the opposition party to hold a vote, the man described that as part of the games he was playing.
"The game is to play this general election thing," he said. Johnson insists Britain must leave the E.U. in 55 days, and says an election is the only way to break the deadlock that has seen lawmakers repeatedly reject the divorce deal on offer, but also block attempts to leave the E.U. without one.
He wants to go to the public on Oct. 15, two weeks before the scheduled Brexit day of Oct. 31, but needs the support of two-thirds of lawmakers to trigger a snap election.
Johnson lost a vote on the same question this week, but he plans to try again Monday.
The Associated Press contributed to this report.