There are the rare few that become aggressive and dont go back once season is over.
We had a male like that and ended up making him is own room. Before he got sick with stomach cancer I barely handled him , except for vet visits and getting him to the tub or out to the suncage.
he would lunge at us, stalk attack chase, you name it and we tried EVERYTHING.
He wasnt like that with everyone , mostly me and my kids but he tolerated my husband for a few minutes here and there.
You said hed been to the vet, Im guessing he ruled out illness , which can sometimes make them agressive,,,