Just in case you don't know this: iguanas are natural loners except for breeding, so he doesn't actually need much company. Enriching his environment is good, like giving him TV to watch or a window to look out of.
I think all the igs I had held on for dear life. Try holding him up high, it gives some of them confidence. or, wrap him in a large towel when carrying him.
Can you move him into your main living area? Noisy or not, it would improve your relationship.
His love aggression is just normal, not much you can do. But don't just submit to it. Carry a broom to push him away. Slap him with if he's too persistent or if if he's on carpet. He's tougher than you probably realize. Squirt him with cold water to brush him off. Try creating a warm spot for distraction where you have him roaming.
I hope it wasn't a hard bite. But even a soft one can cut like tiny razors :-(
Hmmm....about the fear you feel, does it seem like he lunges or wants to bite hard? That's not love biting. If it's not, I need to know very clearly to give you real advice. Please describe all the various biting he has done - was any of it sudden with destructive head-twisting deep biting? Does his tail ever swish when he gets ready to bite?
If you don't trust him, definitely roll him up in a towel like an iguana burrito before you carry him. Wear 8 inch high boots, and find on-line those inexpensive Kevlar knit gloves and thumb-hole sleeves. they stopped all the bites the nasty igs gave to me. Also wear stiff floor length jeans. Doing all this is a hassle but it will likely improve his attitude toward you if he views you a skittish female love object, or as an interloper he can intimidate.
Try going to him with loves before he decides to go to you. Throw a towel over him, lift it enough to rub his neck,, scratch his crest, hold juicy treats on a fork at his nose, etc.