I think adopting a tried and true adult is the best way to get another iguana.
I'm sure you've known this for years: I just want to add that maybe 95% of iguanas, when re-homed, go through Relocation Stress. It lasts for a week to months. The symptoms are exactly the ones you describe about your iguana. But most pull out of it very well.
Does he twitch or swish his tail, or gape at you? You know what that means :-). IMO for him, this is all perfectly normal for now. A good cautionary move is to make sure he never gets a bite on you. Your autonomic "jerk back" could teach him that biting is an easy way for him to keep you distant.
I'm confident that he will come around soon! You of 3% of all new-ig owners have the vital skills needed!