Our suspicions of Annuneryn rise from the disgrace of his father. We don't know what his father did to be disgraced, nor do we know how they affected Annuneryn.
We also know that he is a self-made Lord, so to speak, having established the title on his own rather than gaining it from his father. Right? We could explore how he was able to do so.