Entity Framework removes too many entries - C#

I'm freaking out with Entity Framework. It should be easier and faster but it's just horrible.

I'm using database first approach with two very simple tables. The User table with Id and Username, primary key and autoincrement on Id, and the Permission table with permissionname and userid, primary key on both columns and a releation from userid to user.id.

This is my code to grant and revoke permissions:

   public bool ContributionMarginCustomer
    {
        get => GetPermission(_contributionmarginCustomer);
        set => SetPermission(value, _contributionmarginCustomer);
    }

The ContributionMarginCustomer property is bound to a checkbox.

private void SetPermission(bool permissionIsGranted, string key)
{
    var permissionStatus = permissionIsGranted ? PermissionStatus.Granted : PermissionStatus.Revoked;

    using (var entity = new KundeninfoEntities())
    {
        var user = entity.Users.Single(x => x.Id == _user.Id);
        var existingPermission = user.Permissions.SingleOrDefault(x => x.Name == key);

        switch (permissionStatus)
        {
            case PermissionStatus.Granted:
                if (existingPermission == null)
                {
                    user.Permissions.Add(new Permission { Name = key });
                    entity.SaveChanges();
                }
                break;

            case PermissionStatus.Revoked:
                if (existingPermission != null)
                {
                    entity.Permissions.Remove(existingPermission);
                    entity.SaveChanges();
                }
                break;
        }

        _permissions = entity.Permissions
                             .Where(x => x.UserId == _user.Id)
                             .ToList();
    }
}

Granting a permission works great.

Removing the permission with entity.Permission.Remove(existingPermission) removes it from every single user in the database.

I do not understand this behaviour. Do any of you?

Thanks

EDIT: Usertable

Permissiontable before:

Permissiontable after I remove it with userId 15 and key CONTRIBUTIONMARGINCUSTOMER

Edit 2: SOLUTION

I changed the permissiontable to have only one column as primary key. That means the new permissiontable has three columns: Id (autoincrement), Name and UserId with a unique key to name and userid.

The code above works now but I'm not really happy with this.