About a hundred years after the Ascension of our Lord into heaven, questions began to be raised about his so-called 'brothers and sisters', mentioned in the Gospels.  None of the Fathers of the Church taught that these were children of the Virgin Mary, and some, like St. Jerome, wrote works defending her perpetual virginity with great energy against the few heretics who dared to question it.  Following is a comment he makes on a text of Matthew 12, 46-50:

" 'While he was still speaking to the crowds, behold, his Mother and brothers were standing outside, seeking to speak to him.  A certain man said to him: Your Mother and your brothers are standing outside, seeking you.  And, responding to the one who spoke to him, he said: Who are my Mother and my brothers?  And he extended his hand over his disciples.'  The Lord was occupied with the work of the word, with the teaching of the people, and with the office of preaching, when a certain man announced that his Mother and brothers came and were standing outside, seeking to speak to him.  It seems to me that this man did not announce this simply and casually, but that he was laying snares for the Savior, to see whether he preferred flesh and blood to the work of the spirit.  The Lord did not deny that he had a Mother and brothers, refusing to go out to them, but he responding to the one seeking to ensnare him when he stretched out his hands over his disciples and said: 'Behold my Mother and my brothers!  For, whoever does the will of my Father in heaven is my brother, sister, and mother.'  These are my mother, who daily bear me in the souls of believers.  These are my brothers, who do the works of of Father.  He did not deny his Mother . . . but he preferred his Apostles to his relatives, so that we ourselves may also prefer the spirit to the love of the flesh.

" 'Behold, your Mother and your brothers are standing outside, seeking you.'  Following the nonsense of apocryphal works and the inventions of a certain Melcha or Escha, the 'brothers' of The Lord were the sons of Joseph by another wife.  As we wrote in our book against Helvidius, the 'brothers' of The Lord were not the sons of Joseph but were our Savior's cousins.  We understand that the children of that Mary who was said to be the mother of James the lesser, Joseph, and Jude, were the children of his Mother's sister.  These are the 'brothers' of The Lord, whose names are given in a passage in another Gospel.  All the Scriptures say that these were his cousins."