CDU won 33 percent of the vote, gaining around 6 percentage points since the last election, while the SPD won about 31.5 percent, having lost nearly 8 percent point since 2012.
North Rhine-Westphalia has been governed by a center-left coalition of the SPD and the Greens since 2010, with the Social Democrat Hannelore Kraft serving as state premier. And Sunday's election in the country's most populous state is widely seen as a key indicator of the national mood ahead of national elections in September.
Projections for ARD and ZDF public television, based on partial counting, showed Merkel's Christian Democratic Union beating the Social Democrats by 33 percent or more to around 31 percent.
The conservatives, who have now defeated the SPD in three state elections since the end of March, are hoping to build on the momentum in the run-up to the federal vote on September 24, where Merkel is seeking a fourth term.
Following the result, Mr Laschet will discuss forming a coalition with the Free Democrats, which picked up 12.6% of the vote.
But the results marked a huge blow to Martin Schulz, who had generated a strong surge in support for the SPD when he took over the party in February.
In the North Rhine-Westphalia campaign, Mrs Merkel's conservatives sought to portray Mrs Kraft's government as slack on security, and also assailed what they said is regional authorities' poor handling of education and infrastructure projects.
"So, today we have an occasion to celebrate, but tomorrow we start serious work for the benefit of Germany", Secretary General of the CDU Party Peter Tauber told journalists in Berlin.
Energized Ducks even Western finals, beat Predators 5-3
This must mean they took more accurate shots and/or the Ducks defense was allowing the Predators to take many shots on goal. Nashville had the past four days off after eliminating the Blues to cap its 8-2 surge through the first two rounds.
Audi, Volvo models will soon run a full version of Android
Bringing the Assistant to iOS certainly can't hurt, given how many Google services already populate the average iPhone in the US. Volvo has agreed a deal with Google to develop the next generation of its infotainment and connectivity system.
Jose Mourinho already picking Manchester United players for Europa League final
Tottenham will have to get over their Wembley jitters to do that, but the future certainly looks bright as their new stadium takes shape.
Political scientist Oskar Niedermayer told dpa before the state election that if the SPD came in second, then Schulz "can bury his hopes for being chancellor".
The win by Merkel's Christian Democratic Union (CDU) in North Rhine Westphalia built on the CDU's impressive victories in the two other recent state elections - in the coastal region of Schleswig Holstein and the small industrial state of Saarland. It was still unclear late Sunday whether the far-left Die Linke would reach the 5 percent threshold to enter the state parliament. About 13 million of North Rhine Westphalia " s 18 million residents were eligible to vote, representing more than one-fifth of the total national electorate.
"Incumbents are certainly not safe - incumbents can lose", said Thomas Oppermann, the party's national parliamentary leader. "I am convinced that for the past seven years, step by step, we have helped this state move forward", Kraft said in the state capital, Duesseldorf.
The result was the Social Democrats' worst in the region since World War Two, with CDU deputy chairman Armin Laschet set to become the new governor of North Rhine-Westphalia. In the state's last election in 2012, the Social Democrats beat the CDU by 39.1 percent to 26.3 percent.
The party has led governments in the state for 45 of the past 50 years.
"The issue of fairness is, of course, very important but I am convinced that the Social Democrats are struggling with the concept of innovation and are getting things the wrong way round", she said.
North Rhine-Westphalia was the last state to hit the polls before the general election September 24.