How does Hamlet change over the course of the play as in how does he become more self-aware?I thinking of something like how he thinks too much and...

The question of how Hamlet's self-awareness develops in Shakespeare's Hamlet is extremely complex, and very much open to interpretation.  That makes it kind of fun.  But you're likely to get a variety of answers, and no one answer will tackle all of the angles. 

The player's speech in Act II:ii serves as a rebuke to Hamlet.  The actor reveals such powerful emotion even though he is just acting a "fiction" that Hamlet is humiliated because he, unlike the actor, is in the center of an actual situation, but has so far done nothing.  The actor had "tears in his eyes" and a "broken voice."  His entire body fit the role he was playing.  Hamlet, reacting to this performance, calls himself a "rogue and peasant slave."   He is a "dull and muddy-mettled rascal," and he asks:  "Am I a coward?"

One can certainly see this scene as contibuting to Hamlet's self-awarness.  Yet, again, the issue is complex.  By the end of that very speech Hamlet raises the question of the identity of the ghost who earlier told Hamlet how his father was killed.  As long as the identity of the ghost is in question, Hamlet cannot be sure as to Claudius' guilt.  He reveals a plan to have the actors portray a death scene that features a murder performed in a similar way to how the ghost says his father was killed.  If Claudius overreacts, then Hamlet will know for certain that Claudius is guilty.  Hamlet says:

...I know my course.  The spirit that I have seen

May be a devil, and the devil hath power

T' assume a pleasing shape, yea, and perhaps

Out of my weakness and my melancholy,

As he is very potent with such spirits,

Abuses me to damn me.  I'll have grounds

More relative than this.  The play's the thing

Wherein I'll catch the conscience of the king.

So you see, within the same speech Hamlet seems to rebuke himself for his inaction, but also raises an important question:  is the ghost a demon sent to abuse him (as the witches do to Macbeth in Macbeth, for instance), or is the ghost truly the spirit of his father?  Hamlet's revenge depends on that question.  He does not want to kill an innocent Claudius, only a guilty one.

Hamlet's self-awareness also develops in the gravedigger scene, the Fortinbras scene, etc.  I have just presented an analysis of one speech for you.  I'll let someone else handle the rest.