They suspected Jim had planned to destroy Oliver.

They were angry because Jim was cruel to his biological father for his personal interests.

But he wasn't for his personal interests because he didn't care about the Brown Group at all.

Besides, Oliver was not his biological fathe