Delete agent from a Workitem

How to delete Actual agents from a Workitem without effecting other users who have received the same workitem. There are 9 Actual Agents for the workitem who has received the workitem. I need to remove a user from the workitem Actual Agent list.