Well well, what can one speculate? No rumours available. Hahaaaaa..
Maybe he became in the end just too much a pamperer. Not man enough. As for his appearance, he always looked like a car- or insurance salesman of some sorts. Too dandy-ish for my taste. No charisma at all. But I guess he was in a way a good manager for Shirley in the later years of her career, when she did mostly concerts instead of recording. Shirley was always a great cashbox concertwise, so under his authority the big money kept flowing in.