James Whalen was born in Collingwood, Ontario in 1869. In 1875, at the age of six, his family moved to Port Arthur, Ontario. Not long after the Whalens' arrival in Port Arthur, James' father drowned and he was left as the sole provider for his mother and younger siblings.

Once out of public school Whalen entered the logging and railway contracting business with a very strong drive for achievement. He cared deeply for the city of Port Arthur and wished to develop the city as much as possible. The empire which he eventually built included logging, pebble, dredging, shipbuilding, insurance, and real estate companies. For his involvement with the dredging and shipping industries, Whalen came to be known as the man who "put the Port in Port Arthur." One of his great accomplishments was the construction of the Whalen Building in 1913, which still stands today as the Thunder Bay Hydro Building. When it opened in 1914 it was said to be "the finest between Toronto and Winnipeg."

James Whalen married Laurel Conmee, the daughter of James Conmee, Member of Parliament. Together James and Laurel had five children, Hazel, Edward, Jim, Loley, and
Margaret.

During his last years, Whalen moved to the West coast to continue working with the pulp and paper industries. Within a few years he was diagnosed with Bright's Disease, affecting his kidneys. He passed away on June 4, 1929, while being treated in a Duluth, MN hospital. In recognition of this man and his accomplishments, the city of Port Arthur honoured him with a half-day holiday.