[He's quiet for a long time, though not because he doesn't want to answer. He just can't find the right words to express why it's so important, because he still doesn't understand why he trusts Steve.]

They're connected to SHIELD, and they might want to contain me again. You'd fight them on that, lose them. I don't want that.

[Even if he doesn't know why it bothers him that Steve might lose his friends.]

And I won't be trapped again. If you come with them, then it can't work. Better to let it go.