There’s no need for Modify All Data - we were able to make it work by creating a permission set granting READ and EDIT access to the standard objects mentioned here (and for all custom objects you plan to sync):
- Leads
- Contacts
- Accounts
- Opportunities
- Campaigns
- Tasks *
- Users *
* did not need Tasks or Users, even though it was mentioned in the link.
View All Fields were also granted for our use case, although (while untested by us), I think it should work without.