One of the big ones is they play a large role in steering the fate of the galaxy. Using precog as a guide, they nudge things to make events happen. Mostly this is good for the people, but it can also be viewed as being for the good of the Jedi.
Also, they do sometimes fall to the dark side, and the Jedi can be viewed as the source of the Sith and a lot of large-scale wars as a result. One of the biggest Sith vs Jedi conflicts was called "the Jedi Civil War" even. The differences between the two aren't as obvious to people outside the order who often think Sith just means "Jedi the Jedi don't like".
On a more personal level, Jedi do enjoy a level of privilege/exemption from the rules in most eras. They're sort of like cops without jurisdiction limits, and locals often don't like it when that type show up even in real life.
They also make themselves opponents of fairly influential types- the various guilds that formed the CIS had a whole lot of pull, for example, so plenty of propaganda to convince governments to not like Jedi and tell their people not to was made, and even worlds that didn't buy into the CIS line may still grow more cautious as a result. Or aside from that, a world may have a ruler or such toppled who was a bad guy, but not viewed as such by the people and/or who viewed the Jedi toppling them as outsider meddling. Depending on the history, a lot of worlds will thus distrust them.