Culture Secretary Lucy Frazer has blamed heavy Tory losses in the local elections on the pandemic and the Ukraine war.
She also noted the Conservative Party had been in power for “a long time”.
The Cabinet minister conceded that voters were “frustrated and angry”, but claimed Rishi Sunak was starting to regain the public’s trust.
The Prime Minister has been blamed by some Tories after the party shed 960 councillors in Thursday’s poll.
Ms Frazer told Sky News’ Sophy Ridge On Sunday programme: “If we could look at the context for the whole local election, we’ve been in power for a long time.
“We’ve just had a pandemic which has disrupted many people’s lives and has had consequences for the economy, as has the war in Ukraine, which is going on.”
Pressed on whether she understood the scale of her party’s losses, she said: “Totally, it’s really important that we listen to people. I know people are frustrated and angry.”
She said she recognised the results were “not good”, adding: “I totally understand that we need to do better and I think we are going to deliver.”
The party was rebuilding trust as Mr Sunak was “starting to deliver in a quiet way” his five priorities including curbing inflation, Ms Frazer argued.
“I totally recognise we’ve had a really difficult few years.
“I do think that the Prime Minister, who’s now been in office for six months, is getting the country back on track and is delivering. I think we’re starting to gain the trust of the British public.”
Pressed on whether the Tories needed to offer more than Mr Sunak’s promises from January, she told the BBC’s Laura Kuenssberg On Sunday programme: “We absolutely need to reflect. I think Rishi’s only been the Prime Minister for six months.”
Kuenssberg said: “It’s not five minutes, it’s six months.”
“It’s not five minutes but these are huge challenges,” Ms Frazer replied.
Meanwhile, Labour was celebrating “pretty encouraging” local election results, which saw Sir Keir Starmer’s party gain 635 seats and take control of another 22 local authorities.
Shadow health secretary Wes Streeting told Ridge: “I think for the next general election Labour feels confident but not complacent.”