“What’s frustrating me is I firmly believe that House Republicans are going to lose the majority,” Gonzales told Punchbowl News at the Texas Tribune festival Thursday. “And we’re going to lose it because of ourselves.”

House Speaker Mike Johnson’s (R-LA) office declined to comment. The National Republican Congressional Committee sent a two-word response to the Washington Examiner over Gonzales’s remarks: “We disagree.” The Democratic Congressional Campaign Committee released a statement in a similar fashion: “We agree.”

Gonzales has spoken out against his party in the past. He received a censure from the Texas Republican Party last year after he voted for the bipartisan gun safety law in the wake of the Uvalde school shooting in his district. At the time, the NRCC defended Gonzales. The congressman also voted for a bill codifying protections for same-sex marriage.

Republicans are defending a handful of competitive seats this cycle. Polls and projections show that the House is up for grabs as some vulnerable GOP lawmakers’ districts trend more to the left.

Gonzales’s seat in Texas is considered “solid Republican,” after the congressman defeated strong challenger Brandon Herrera in a runoff in May. The incumbent faced heat from his hard-line Republican colleagues after he said Reps. Matt Gaetz (R-FL) and Bob Good (R-VA) are among the “scumbags” he serves with in Congress.

Both Gaetz and Good endorsed Herrera. While Gonzales defeated Herrera, Good lost his primary election after fellow Republicans endorsed his Virginia challenger, John McGuire.

House Republicans hold a four-seat majority after a series of deaths and resignations on both sides of the aisle, meaning Johnson can only afford to lose four votes to pass a measure along party lines. Lawmakers return to Capitol Hill next week to pass government spending bills, with border and China legislation also on the docket.