An Ignorant Kohen Gadol?

Rebbi Yose ben Kisma said: Once I was walking by the way when I met a man. He greeted me and said, “Rabbi, where are you from?” I responded, “I am from a great city of sages and scribes.” He said to me, “Rabbi, would you consider living with us in our place? I would give you a thousand thousand denarii of gold, and precious stones and pearls.” I responded, “My son, even if you were to give me all the silver and gold, precious stones and pearls that are in the world, I would not dwell anywhere except in a place of Torah; for when a man passes away there accompany him neither gold nor silver, nor precious stones nor pearls, but Torah and good deeds alone.” (Avos 6:9)
The commentators wonder: if Rebbi Yose Ben Kisma was offered so much money to relocate, why didn’t he just establish a new makom Torah in the new locale? After all, his only objection was that the man’s city wasn’t a makom Torah. So make it a makom Torah! All of his newly found wealth would be put to great use establishing a yeshiva!