Well.... that's a little too simple. We have cases in my country where pedophiles actually wanted to start a political party to support their identity. They feel children of say 12 years old can very well decide form themselves if they want sex or not. They feel they should be allowed to follow their urge if the child consents.
Plus there's another thing. A pedophile who forces a child to have sex, is only a molester from the POV of the child and its parents. From the pedophile's POV he's just ACTING on his feelings.
A step further... child pornography is also illegal. In my country possession of child porno is already illegal. So a guy who ACTS on his urge of pedophaelia to watch these films is already transgressing the law.
If attraction defines a sexual identity, someone with unacceptable sexual attraction could therefore be condemned for his feelings, if not by law, then by people in his neighbourhood. I mean, if you had children, I'm sure you'd like to know if someone in your neighborhoud has a sexual attraction to children, even if he didn't act on it, has never acted on it and doesn't intend to act on it. But still, by definition there's a pedophile living in your neighborhood.
See, it does get rather complicated. In these cases, the ACT defines someone IMHO, because it reveals a feeling. And that revelation has consequences for the surroundings... not the feelings themselves.