Mijo (edit 3)

[TranquillityBase]

I try to read your face,
are you angry or just serious?


Just does one of two things: makes sound petty, or gets anger and serious mixed up with connotations of justice.
These are the little language games that poetry masters.

The word only would work better. Only insinuates both lonely and the opposite of lonely, being one is enough. 

You try to read the face, but the face is only the face. You don't read it. 
And the serious though not angry adds an emotional commentary on how people interact and perceive each other or fail to. 

I'm missing a piece of the puzzle:  Where would "only" appear in these lines?