Abū al-Walīd al-Azraqī (d. 250 H) in his books of Akhbar Makkah wa ma ja’ fiha min aathaarin (The Reports of Makkah and What Relics are Therein, pg. 158), in the mention of the locations in which it is commendable to pray in Makkah and what lies therein of the relics of the Prophet (صلى الله عليه وسلم) and what is venerable from them cites:
The house, in which the Prophet (pbuh) was born, is the house of Muhammad bin Yūsuf, the brother of al-Hajjāj bin Yūsuf. ‘Aqīl took the responsibility at the time the Prophet (صلى الله عليه وسلم) migrated. The Messenger of Allāh (صلى الله عليه وسلم) says on the year of the Farewell Speech, when it was said to him, “Where shall we live O Messenger of Allāh (صلى الله عليه وسلم),” asked “has ‘Aqīl left for us a shade (shelter or house)?” And the house remained in he and his son’s custody until his son sold it to Muhammad bin Yūsuf after which he moved in and is call ‘al-Baydā,’ which is known [even] today as “ibn Yūsuf.” And this household remained until (the Abbasid queen) al-Khayzurān (d. 173 AH), mother of the two khalīfs, Mūsa and Hārūn made hajj and changed it into a place of worship and evicted them from the house and bought it from the foundation to the alley which runs by the house, for which it is called the Alley of the Birthplace of the Prophet (صلى الله عليه وسلم)
Muhammad bin Jarīr at-Tabarī (224-320 H) said is his Tārīkh, from Ibn Ishāq relates:
The Messenger of Allāh (صلى الله عليه وسلم) was born on Monday in the year of the Elephant. And it is said that he (pbuh) was born in the house related to ibn ‘Aqīl. Indeed the Messenger of Allāh (صلى الله عليه وسلم) granted it to ‘Aqīl bin Abī Tālib and it did not leave the care of ‘Aqīl until his death. His son sold it to Muhammad bin Yūsuf, the brother of al-Hajjāj bin Yūsuf, who sold it and built his house to what it is referred to as ‘the house of ibn Yūsuf’ and this household remained therein until al-Khayzurān ejected them and made it into a place of worship.
[Muhammad bin Jarīr at-Tabarī, Tārīkh al-umam wal-muluk (History of Nations and Dynasties), p. 156.]
According to Wahhabis, although they claim to follow the salaf, they say: “It is not permissible for Muslims to build Masjids or to offer Salah (Prayer) at the places trodden by the Prophets as this is a means to Shirk (associating others with Allah in His Divinity or worship).” [Fatwa of Ibn Baz]