Joshua Boyle spoke to reporters after the family returned to Canada.
Taliban spokesman Zabiullah Mujahid rejected that as propaganda by the Western governments that helped rescue the family.
Boyle also expressed his displeasure with U.S. foreign policy by gesturing to one of the U.S. State Department officials and saying, "Their interests are not my interests".
"We join the Boyle family in rejoicing over the long-awaited return to Canada of their loved ones", the Canadian government said Friday.
The family was rescued Wednesday on a Pakistani-led raid based on US intelligence.
Boyle said his wife was raped and one of their children was killed during their time in captivity..
Washington considers the Haqqani group a terrorist organization and has targeted its leaders with drone strikes.
Pakistan's military said the family were rescued after entering Pakistan from Afghanistan, but two Taliban sources with knowledge of the family's captivity said they had been kept in Pakistan in recent years.
"God has given me and my family unparalleled resilience and determination, and to allow that to stagnate, to pursue personal pleasure or comfort while there is still deliberate and organized injustice in the world would be a betrayal of all I believe, and tantamount to sacrilege", he wrote. Coleman was pregnant at the time.
Aaron Rodgers broken collarbone means the Steelers miss him again
Matt Flynn , the quarterback who kept the Packers afloat in 2013, most likely isn't walking through the door. The Packers announced a shoulder injury and listed Rodgers as questionable to return.
Ezekiel Elliott Requests Full-Panel Rehearing
The original punishment was issued by NFL commissioner Roger Goodell after a 13-month league investigation. McFadden will have even more gas in the tank after being inactive for the first five games of the year.
Twitter CEO Jack Dorsey promises tougher stance on abusive tweets
The new rules on abuse and hate tweets will start rolling in the next few days and more information on the same will be shared . Twitter routinely promises to fix it , and routinely has to promise to "be better" a few months later.
"Taking your pregnant wife to a very unsafe place to me and the kind of person that I am is unconscionable", Jim Coleman, father of Caitlan Coleman said.
"The stupidity and the evil of the Haqqani network's kidnapping of a pilgrim and his heavily pregnant wife engaged in helping ordinary villagers in Taliban-controlled regions of Afghanistan was eclipsed only by the stupidity and evil of authorizing the murder of my infant daughter, Martyr Boyle", Joshua Boyle told reporters upon his arrival Friday night at Toronto's Pearson International Airport.
The Canadian, his wife and their children were intercepted by Pakistani forces while being transported in the back of their captors' auto.
Over the weekend however, Boyle accused his captors of having killed their infant daughter and of raping his wife. The team did a preliminary health assessment and had a transport plane ready to go, but sometime after daybreak Thursday, as the family members were walking to the plane, Boyle said he did not want to board, the official said.
Two US officials, speaking on condition of anonymity, on Thursday told Reuters that the USA military had been ready to fly the family out of the country after they were freed but said Boyle, who is Canadian, had refused to board the aircraft.
He said Pakistani troops and intelligence agents, acting on a USA intelligence tip, zeroed in on a vehicle holding the family as they were being moved in Kurram agency.
Boyle previously had been married to the sister of Omar Khadr, a Canadian who spent 10 years at Guantanamo Bay after being captured in 2002 in a firefight at an al-Qaida compound in Afghanistan. They divorced in 2010. Officials had discounted any link between that background and Boyle's capture, with one describing it in 2014 as a "horrible coincidence".
The US official said there were some questions surrounding Boyle's past, but the Department of Justice said he did not face arrest.