The British comic moved with his wife Julia Carey and their family to Los Angeles five years ago to present the nightly programme when he succeeded Craig Ferguson as the host of the show in March 2015. But speaking to Britain's Daily Mail newspaper, James suggested family commitments may force him to step down from the show.
"I have a year left on my contract. It’s hard. It’s so much more than just, 'What do I want to do?'," the 40-year-old reflected. "We’re a long way from home, my wife and I, and our children, and there are people at home that we miss deeply and we care about and they’re getting older.
"Then there is the question of where do we educate our children and where is the best environment for them." The star went on to insist the conflict wasn't about money, and claimed he doesn't want to get too comfortable in the role.
James and Julia share three children - son Max, eight, and daughters Carey, four, and 18-month-old Charlotte.