The first settler in the Lithgow-Wallerawang district was James Walker. He was born in Perth, Scotland, in 1785, the son of wealthy merchant. He became an artillary officer in the Royal Marines and retired in 1822. In May, 1823, he set sail on the Brutus for N.S.W. Another young Scot accompanied him by the name of Andrew Brown, then aged 26.

The first clergyman to reside and work in Lithgow was Rev. Colin Stewart. He was a Scot, like so many of the early settlers in the Lithgow District, and was born in 1803. He was ordained a minister in 1838 and arrived in N.S.W. in 1839. He arrived and preached his first sermon at Hartley in 1839.

Perhaps the man with the best claim to be the founder of Lithgow was Thomas Brown, born in Scotland in 1811. He was the second in a family of eleven. He and his wife (five years older than himself) set sail from Liverpool for Sydney in 1838.