There's an intentional affinity among Graham, Dollarhyde, and Lecter, in that they have many of the same traits but occupy different points on the moral spectrum.

All three of them are extremely intelligent, introverted, and potentially dangerously violent. Graham takes his gifts and his darker side and channels them on behalf of the law and conventional morality. Dollarhyde is unable to restrain his darker side, he struggles against it and fails. Lecter entirely embraces his darker side and revels in it, telling Graham "We don't invent our natures, we inherit them along with our internal organs..."

Lecter recognizes his affinity with Graham and with Dollarhyde, and successfully eggs both of them on along their paths. Similarly, there's a scene that didn't make it into the director's cut where Graham acknowledges empathy for Dollarhyde.

These affinities are the centerpiece of the movie, which is why Noonan's portrayal of Dollarhyde was so essential: he was somewhere between Lecter and Graham on on the nihilism vs. morality scale, and so could understand and be understood by both.