On the authority of Ibn Abbas (r.a) , Rasulullah (SAW) said : "Anyone who speaks on Jum'uah day while the Imam is delivering the khutba, his similitude is like a donkey carrying books, and the one who says to him be quiet, is guilty of speaking during the sermon and thus he the speaker has no Jumu'ah" (Musnad Ahmad vol 2, p 244). In another narration by Abu Hurairah (ra), Rasulullah (SAW) said : "when the Imam is delivering the Khutba and you ask your companion to keep quiet and listen then no doubt you have done an evil act." (Sahih Bukhari vol 2. Pg 27 no 56) Hence the sin of speaking during the sermon on Friday is so grave to the extent that it renders your Jumu'ah null and void. Even to say to your brother keep quiet is impermissible. May Allah save us all.