It is said that Abraham (sws) stood on this very stone when he raised its foundations.Azraqi, Akhbar Makkah, vol., 1, p. 59
When Muhammad attacked Mecca and won the Quraysh tribes, he entered the Ka'aba and destroyed every icon or sculptured idol, both pagan gods and after some hesitation also Christian icons(?) of Jesus, Mary and Abraham*.
[After the conquest of Mecca] "Apart from the icon of the Virgin Mary and the child Jesus, and a painting of an old man, said to be Abraham, the walls inside [Kaaba] had been covered with pictures of pagan deities. Placing his hand protectively over the icon, the Prophet told `Uthman to see that all other paintings, except that of Abraham, were effaced." (Martin Lings, "Muhammad: His Life Based on the Earliest Sources" p.300, ref: al-Waqidi, Kitab al-Maghazi 834, and Azraqi, Akhbar Makkah vol. 1, p. 107. Martin Lings is a practicing Muslim.)
"... pictures of the prophets and pictures of trees and of angels. Among them there was a picture of Ibrahim as of an elderly man, drawing lots with arrow lots, and the picture of Jesus, the son of Mary, and of his mother and a picture with angels." (I quote al-Azraqi according to the Arabic text edited by Ferdinand Wuestenfeld, Chroniken der Stadt Mekka, Band 1, Leipzig 1858, reprint Beyrouth 1964, p. 110 s. There is, to my knowledge, no translation into an European language.)
Hisham ihn-Muhammad al-Kalbi said: I was informed by my father and others, and I personally checked and ascertained their report, that when Ishmael, the son of Abraham, settled in Mecca, he begot many children. [Their descendants] multiplied so much that they crowded the city and supplanted its original inhabitants, the Amalekites. Later on Mecca became overcrowded with them, and dissension and strife arose among them, causing them to fight among themselves and consequently be dispersed throughout the land where they roamed seeking a livelihood.
The reason which led them to the worship of images and stones was the following: No one left Mecca without carrying away with him a stone from the stones of the Sacred House (al-Haram) as a token of reverence to it, and as a sign of deep affection to Mecca. Wherever he settled he would erect that stone and circumambulate it in the same manner he used to circumambulate the Ka'bah [before his departure from Mecca], seeking thereby its blessing and affirming his deep affection for the Sacred House. In fact, the Arabs still venerate the Ka'bah and Mecca and journey to them in order to perform the pilgrimage and visitation, conforming thereby to the time honored custom which they inherited from Abraham and Ishmael.
In time this led them to the worship of whatever took their fancy, and caused them to forget their former worship. They exchanged the religion of Abraham and Ishmael for another. Consequently they took to the worship of images, becoming like the nations before them. They sought and determined what the people of Noah had worshiped of these images and adopted the worship of those which were still remembered among them. Among these devotional practices were some which came down from the time of Abraham and Ishmael, such as the veneration of the Houseand its circumambulation, (Book of Idols, Page 4).
10. Muhammad ibn-al-Sa'ib al-Kalbi, d. A.H. 146 / A.D. 763; al-Fihrist, p.95.
11. The Ka'bah.