best western case study

Customers Weigh In on Best Western Through New Survey System

Half a million customers weigh in on best western through new survey system.

PHOENIX | Best Western's nearly 4,200 hotels around the globe are getting advice on how to run their business from an important audience: customers. Today, the brand's new customer feedback solution, Medallia, received its 500,000th completed survey, after implementing the system just six months ago.

"Having customers tell us what we're doing right and wrong helps us to enhance the Best Western experience for the nearly 400,000 guests that stay with us each night, " said Ric Leutwyler, Best Western International's senior vice president of brand quality and member service. "Our mission is nothing less than to lead the industry in customer care, and the Medallia guest survey is a major step toward that goal."

After a guest stays at a Best Western property, Medallia sends an e-mail with links to an online survey. Once the survey is completed, the data goes directly into the Medallia customer experience management system, allowing properties to track guest satisfaction and identify any problem areas that need to be addressed.

"We are thrilled to join forces with Best Western International, the world's largest hotel chain," said Elizabeth Carducci, head of Medallia's hospitality practice group. "Medallia's program for Best Western is probably one of the single largest global integrated programs - certainly in the hotel industry - with deployment in 78 countries and more than 18 languages."

Leutwyler noted that a customer-focused culture of care will differentiate Best Western International from other hotel chains in the coming years, and that Medallia is a critical partner in this endeavor.

ABOUT BEST WESTERN INTERNATIONAL | Best Western International is THE WORLD'S LARGEST HOTEL CHAIN®, providing marketing, reservations and operational support to 4,200* independently owned and operated member hotels in 80* countries and territories worldwide. Founded in 1946, this iconic brand is host to approximately 400,000 worldwide guests each night. A pioneer within the industry, Best Western is recognized for its distinctive business model and diverse hotel portfolio. The company continues to innovate and enhance both the business and leisure travel experience. Since 2004, Best Western has served as the Official Hotel of NASCAR®. For more information or to make a reservation, please visit .

*Numbers are approximate and can fluctuate.

ABOUT MEDALLIA | Medallia (), founded in 2001, provides enterprise feedback solutions to Global 2000 companies. More than 25,000 businesses and business units around the world use the Medallia system to track customer satisfaction. Medallia¿s solutions enable companies to gather, monitor, and act on feedback from customers, partners, and employees. Customers include global hotel, financial services, retail, and high tech companies. The company is headquartered in Silicon Valley (Menlo Park, Calif.).

Your content on Hospitality Net?

Hospitality Net membership explained

22:1 return on investment

Best Western represents over 4,000 independent hotels worldwide and 280 in the UK. Each hotel has to meet strict quality controls but they are free to express themselves and run their business as they see fit. In the UK Best Western was struggling for identity. Brand tracking showed a mediocre and stagnant performance whilst the budget and luxury sectors expanded. Stuck in the middle of the road and in danger of getting run over by the budget hotel train. People weren’t talking about Best Western!

280 hotels, from 12th century castles, to modern city centre hotels makes consistency a challenge. This is exacerbated by the fact that the people working in the hotels aren’t employed by Best Western.

Our approach

Define the target customer.

Working with the data and detailed insight we were able to define the target customer. The growth of budget hotel brands was creating a sea of sameness. It wouldn’t matter which town, city or region you were in, every hotel looked and felt the same. For those independent mind travellers this corporate box approach didn’t appeal. This became the target customer of Best Western

Re-position the brand

A new position focused on celebrating independence and all the amazing hotel stories was created. Stories of hotels with their own vineyard to stories of great service were all brought to life in the new strapline. Hotels with personality. This was more than a new strapline, it was a whole new way of thinking. It was about celebrating the independence of the hotels and differentiating Best Western from all other hotel brands. ‘Hotels with personality’ had to be more than just a strapline it had to be delivered through the customer experience.

Design and implement the customer experience

We designed and implemented the ‘personality experience’. This included:

• Designing the optimum customer journey
• Defining the customer promise
• Introducing a strategy focused on personalisation across digital and physical touch points.
• Training thousands of people who worked across Best Western in delivering the personality experience. This was no easy task with 280 independent hotels.
• Measuring the commercial impact of the experience

Results achieved

The whole strategy created real value for the brand, customers, hotels and delivered a significant commercial return. Result highlights:

• 22:1 Return on investment
• £11m incremental revenue growth in year 1
• Net Promoter Score increased by 7.4%
• Employee engagement increased 346%
• 91% increase in Brand Index
• Trip Advisor comments improved

Get in touch if you’d like a return on your customer experience

Your message

Sign me up for the latest behavioural science news

Best Western

At best western, even corporate it can get a good night’s sleep.

“We have reservations coming in through the internet, through third party travel agencies, as well as local traffic. In centralizing that information, which includes credit card and PII [personally identifiable information], we had to make sure we had the appropriate security processes in place. Having a global solution was important to us.”

– Harold Dibler,  Vice President of Technology, Best Western Hotels and Resorts

Best Western Hotels and Resorts (BW) is an internationally recognized hospitality chain, with 2,200 properties in the U.S. and about 4,000 around the world. Because each BW property operates independently, managing network security was a challenge. After noticing an increase in social engineering at the hotels—and varying levels of success in combatting the issue—BW directed all hotels in North America to deploy the Fortinet Security Fabric. In doing so, the affiliate hotels gained a host of additional threat protection features, alongside the traditional firewall protection.

Keeping Hotels Independent and Secure

“We require [affiliates] to be compliant, but we have to be more proactive, making sure we enforce it, as opposed to just auditing it. Also, we need to be able to talk to our executive management board to make sure they understand the business impact, not just the technology.” -- Harold Dibler, Vice President of Technology, Best Western Hotels and Resorts

Business Impact

Reduced risk of data privacy compromise

Expanded threat protection for individual hotels

Increased ability to enforce compliance with security standards

Improved insight into security posture for corporate management

Related Stories

Rapidly Deployed Secure Networking Supports Fast Growing Quick-Service Restaurant Group

Brazilian Beach Resorts Boost Wi-Fi Connectivity and Network Security With Fortinet Solution

Luxury Hotel Company Enhances Guest Experience

Learn More About the Solutions

Next-Generation Firewall

Quick links.

Free Product Demo

Explore key features and capabilities, and experience user interfaces.

Resource Center

Download from a wide range of educational material and documents.

Free Trials

Test our products and solutions.

Contact Sales

Have a question? We're here to help.

Customer Care Improvements at Best Western

The challenge.

In 2016, Mursion partnered with Best Western® Hotels and Resorts to help one of the world’s largest hotel chains reach its vision to “lead the industry in superior customer care.” After field testing the Mursion virtual simulation platform, Best Western integrated simulations into its “I Care – Every Guest, Every Time” program, a site-based training program designed to improve guest interactions throughout the guest’s hotel stay.

The Solution

Delivered directly to all of Best Western’s North American properties by a team of 42 regional training coaches called regional services managers, the program emphasizes problem resolution, an area of customer service that challenged many Best Western sites according to customer feedback data provided to Best Western by Medallia, Inc. No role faced greater customer service challenges than that of front desk clerks, who have to manage the short tempers and high demands of tired and often frustrated business travelers every day. Best Western’s front desk training included the following:

• Training Module: Each front desk staff member participates in a module that is delivered by a regional services manager focused on how to implement Best Western’s service standards, while going above and beyond the call of duty for every customer.
• Live Simulation: Each front desk staff person participates in 1-2 live virtual simulations with Mursion, in which they interact with avatar-based characters that present challenging issues that mimic real-life customer problems.
• After-Action Review: Immediately following the simulation sessions, regional service managers and general managers deliver standards-based feedback to front desk staff, and the team reflects on how to better handle similar problems moving forward.

“Results from the program are staggering. Hotels that received the training experienced the highest short term gains in customer satisfaction that Best Western has ever measured in such a short period of time.”

– Bruce Weinberg, VP of Operations at Best Western

Customer Benefit

Initial results of the program based on the first cohort of 380 hotels who received this training, demonstrate that Mursion is delivering on its promise to transform customer service. As Best Western recently reported, “Results from the program are staggering. Hotels that received the training experienced the highest short-term gains in customer satisfaction that Best Western has ever measured in such a short period of time.” The average cost to design and deliver the simulations driving these extraordinary outcomes was less than $165 per hotel.

• Hotels experienced an average of 2-5% gains in post-stay guest satisfaction survey ratings compared to flat rates for non-participating hotels.
• Gains were strongest for problem resolution (5.1%), the main focus of the simulation-based training. 97% of hotels reported being highly satisfied with the training.
• Best Western credits Mursion for helping them to sweep the upper midscale in midscale brands in nearly every customer service category, including “helpful and courteous service” (Business Traveler News, 2017).

Based on the success of the first phase of the project, Best Western and Mursion are currently planning a second phase of training that will include simulations for front desk staff on how to better serve Best Western Rewards members. The front desk agents “actually are being tested while learning in a much more fun environment,” said Best Western Hotels & Resorts Chief Marketing Officer and SVP Dorothy Dowling.

Results from the program are staggering. Hotels that received the training experienced the highest short term gains in customer satisfaction that Best Western has ever measured in such a short period of time. Bruce Weinberg, VP of Operations Best Western

Subscribe for the latest Mursion articles and updates.

By clicking the sign up button above, you consent to allow Mursion to store and process the personal information submitted above to provide you the content requested. View our Terms and Conditions.

Related Case Studies

Ericsson Fosters Inclusive Behaviors through Immersive, Science-Based Learning Experiences

Starting in 2019, Ericsson’s global talent management team sought to...

Dow Pioneers a New Inclusive Leader Approach with Mursion

90% of leaders feel more engaged, motivated, and better equipped...

Experiential Training & Competency-Based Assessments

The Challenge TennCare, Tennessee’s statewide Medicaid program, enlisted Mursion to...

Privacy Overview

Category: Best Western Feature

Date archives, case study best western hotel brussels south & hotel arlon.

Find out how these Benelux hotels have improved their operations, gained better insights and are enjoying local support since switching to Guestline Read more

Best Western Asia plans for 2021 – interview with Rodney Simpson – Part 2

As the biggest PMS supplier to BW Asia hotels, we hear from Rodney Simpson on his thoughts for next year, new openings, contactless technology, room rates and the efficiency of cloud based systems. Read more

Why Guestline are in over 70% of Best Western Asia hotels- interview with Rodney Simpson- Part 1

Guestline's relationship with Best Western Asia has gone from strength to strength in recent years. We spoke to Rodney Simpson to find out how they've adapted their strategy and why they consoldated their PMS suppliers. Read more

Case Study The Dean Court Hotel BW Premier Collection, York

‘Guestline cloud PMS is more effective, and it saves us time and money. It’s fully integrated which makes the whole process more seamless and it does everything we need it to do.' Read more about how Dean Court are working with Guestline here. Read more

Best Western SureStay Plus Hotel distribution Case Study

With an integrated PMS and Channel Manager, this Bangkok hotel now benefits from more simplified rate management and on-site support from the local team which has been invaluable. Read more

Best Western Passage House Hotel

The new systems have had a huge impact on their rates. They can now change a whole month’s worth of rates in under 20 minutes and have increased their ARR by £6. Read more

Fill out the below form with your query and a Guestliner will be in touch with you soon!

England, Scotland & Wales

Freephone: 0800 413557 Sales: +44 1743 282300 Sales for clients: +44 1743 282304 Support: +44 1743 282700

Ireland & Northern Ireland

Support: +353 1 691 7484 Office: +353 1 574 7631 Sales: +353 87 28 8987 6

Germany, Austria & Switzerland

DACH Sales: +49 89 541980360 DACH Support: +49 89 380 388 40

Switzerland Sales: +41 41 5620 542 Switzerland Support: +41 41 5620 504

Belgium, the Netherlands & Luxembourg

Sales: +31 40 798 7204 Support: +31 40 798 7256

Asia-Pacific

Office APAC: +66 (0)22079 222

Support Thailand: +66 (0)254 40227 Support Rest of Asia: +63 (0)223 12526 Support Australia: +61 (0) 2720 84478

Make a Support Enquiry

Don't forget, answers to many frequently asked support queries can be found on our Support Portal

For a guide on how to make a Support Enquiry please visit our Support Portal

Privacy Overview

• Capabilities

Best Western International

Faced with fiercely competitive challenges, Best Western management was concerned that their identity system was not in keeping with the reality of the organization or supportive of its repositioning objectives and strategy. Research showed that consumers described the brand as “trucker motels” or “renovated family resorts.” Younger travelers and frequent business travelers who represented Best Western’s growth markets were not interested in staying in the organization’s hotels. Best Western’s identity was costing it business, in spite of its $100 million annual communications budget and sizable investments by its members in design and facility upgrades.

Best Western UX Case Study self.__wrap_n!=1&&self.__wrap_b(":R4klb396:",1)

This is a case study of Best Western’s e-commerce user experience (UX) performance. It’s based on an exhaustive performance review of 193 design elements. 249 other sites have also been benchmarked for a complete picture of the e-commerce UX landscape.

Best Western’s overall e-commerce UX performance is broken. This is mainly due to broken Customer Accounts , Site-Wide Features & Design , and Travel “Booking” Search performances.

First benchmarked in April 2022 and reviewed once in June 2024.

Performance :  -3.1 Broken

URL :  bestwestern.com

Get Premium Research Access

Desktop Web

193 Guidelines · Performance:

Homepage & Main Navigation

9 Guidelines · Performance:

Travel "Booking" Search

46 Guidelines · Performance:

Property & Room Detail Pages

45 Guidelines · Performance:

"Booking" Checkout Process

61 Guidelines · Performance:

Customer Accounts

18 Guidelines · Performance:

Site-Wide Features & Design

14 Guidelines · Performance:

197 Guidelines · Performance:

Mobile Homepage & Main Navigation

Mobile Travel "Booking" Search

44 Guidelines · Performance:

Mobile Property & Room Detail Pages

42 Guidelines · Performance:

Mobile "Booking" Checkout Process

66 Guidelines · Performance:

Mobile Customer Accounts

Mobile Site-Wide Features & Design

22 Guidelines · Performance:

To learn how we calculate our performance scores and read up on our evaluation criteria and scoring algorithm head over to our Methodology page.

The scatterplot you see above is the free version we make public to all our users. If you wish to dive deeper and learn about each guideline and even review your own site you’ll need to get premium access .

Best Western’s Desktop Web E-Commerce Design

13 pages of Best Western’s e-commerce site, marked up with 144 best practice examples:

Best Western’s Mobile Web E-Commerce Design

13 pages of Best Western’s e-commerce site, marked up with 150 best practice examples:

Similar Case Studies

Explore similar case studies of Travel Accommodations sites:

19 page designs: desktop, mobile

27 page designs: desktop, mobile

31 page designs: desktop, mobile

21 page designs: desktop, mobile

22 page designs: desktop, mobile

25 page designs: desktop, mobile

35 page designs: desktop, mobile

34 page designs: desktop, mobile

Explore Other Research Content

Every week, we publish a new article on how to build “state of the art” e-commerce experiences — here’s 5 popular ones:

Drop-Down Usability: When You Should (and Shouldn’t) Use Them

Format the “Expiration Date” Fields Exactly the Same as the Physical Credit Card (72% Don’t)

PDP UX: Core Product Content Is Overlooked in ‘Horizontal Tabs’ Layouts (Yet 28% of Sites Have This Layout)

Form Field Usability: Avoid Extensive Multicolumn Layouts (16% Make This Form Usability Mistake)

Form Usability: Getting ‘Address Line 2’ Right

See all 395 articles in the full public archive.

From First Impressions to Lasting Impressions: How Best Western Hotels Transformed Its Video Marketing with Lumen5

The best way to describe my experience with Lumen5 is that it's incredibly intuitive to pick up — even people without training can use it. We're really happy with what we can produce, and we're making content that's super professional. Robert Schaub Marketing Program Manager, Best Western Hotels

ABOUT BWH HOTELS

Over 4700 hotels worldwide.

• Best Western Plus , which maintains higher design standards and more luxurious amenities such as name brand exercise equipment. 
• Best Western Premiere, top-of-the-line hotels with restaurants, meeting spaces, and other high-end amenities. 
• SureStay , economy-branded hotels with multiple variations.  
• Multiple boutique brands, including Vibe, Glow, Aiden, and Sadie. 

The one thing that unifies each of these brand's is the company's commitment to its guests — to providing people with the best stay possible.

Robert Schaub , Marketing Program Manager, Best Western Hotels

As Best Western Hotels' Marketing Program Manager, Robert is responsible for everything from marketing strategies to negotiating vendor contracts. A BrightEdge Certified professional, he holds a Bachelor's degree in Cinematography and Film/Video Production from Collins College, where he graduated Cum Laude. He has worked with BWH Hotels for more than sixteen years to date.

THE CHALLENGE

Checking out of outdated video content ‎.

For the hospitality sector, interactivity is everything . Today's travellers want to know everything about the experience a hotel offers before they commit to a booking. BWH Hotels knows this. For the past several years, Robert and his team have pushed for more interactive marketing materials that both showcase the company's diverse brands and encourage customers to sign up for the Best Western Rewards program.  Alongside photos and text descriptions, the company began deploying interactive 360-degree walkthroughs created in Google Tours roughly four years ago. These walkthroughs quickly grew outdated. Robert and his colleagues realized that they needed a more agile approach. Enter video.

The issue we kept running into was one of efficiency. How do we show off the amenities and white space of each location and ensure all materials are up to date without detracting from any of our brands? Video was the answer, but that came with its own problems. Robert Schaub Marketing Program Manager, Best Western Hotels

Robert initially began creating branded videos in Adobe After Effects. Unfortunately, this proved ineffective at scale, as a single video could take hours or even days to create. Multiply that by the number of rooms, multiplied by the different locations, multiplied again by the different boutique and unique branding - it simply wasn’t sustainable. The marketing team would find itself trapped in a never ending cycle of scrambling to update old content.  Although working with a video production company would solve this time crunch, it would be at a cost that could charitably be called exorbitant, and with no guarantee that Best Western Hotels wouldn't simply receive a cookie cutter video.   How could each Best Western brand’s unique and luxurious features be accurately reflected in a video without taking forever? This called for a top-ranked yet unique solution that needed to offer the whole nine yards - free nights and no blackout dates included please!

We tried working with production companies, but they wanted $60-$70K per brand, and every time we needed to update a video, they were going to charge us more money to do it. It simply wasn't a cost-effective solution at all. We needed something that we could control, a solution that would allow our marketing consultants to go in and create videos with our photographer's photos. Robert Schaub Marketing Program Manager, Best Western Hotels

THE SOLUTION

Taking the stairs no, the elevator.

As BWH Hotels began assessing potential vendors, the company’s CMO recommended Lumen5 after seeing a live video demonstration from a peer. The CMO and Schaub reached out for a referral, and soon afterwards purchased 26 seats for the platform! And they were off to the races, creating videos for every idea they could think of!

Perfect Timing, Perfect Pairing

Best Western Hotels began onboarding and training on July 12 2022 with an incredibly narrow timeline to support its annual convention in October. Lumen5's onboarding specialist worked closely with Robert’s team to make this happen.This meant empowering them to create videos for pre-conference, mid-event, and post-conference marketing, including promotional videos, recaps, and engaging videos for attendees. The company then began sharing videos internally to promote awareness of the platform, connecting with freelancers working for each of its brands.

"Onboarding and adapting Lumen5 to our needs was totally painless. There wasn't much of a learning curve, and they're very supportive of their product and our needs," says Robert Schaub, Marketing Program Manager, Best Western Hotels Lumen5 empowers video production workflows with sophisticated AI (artificial intelligence) and an intuitive drag-and-drop interface. Requiring only an Internet connection, the platform maintains an extensive library of stock assets with full support for branded content. 

Scaling and Maximizing Video Efforts for Eleven Luxury Brands 

Lumen5’s Creative Services team built 4 custom templates in 60 days , covering 11 brands — something that was impossible with the agency model. This special upgrade allowed Best Western Hotels to fully launch their video marketing at scale across its biggest luxury brands.

"The AI aspect of Lumen5 was a big draw for us, allowing the platform to be truly plug-and-play with images and media, especially with branded materials and templates. We were also impressed with Lumen5's extensive and intuitive customization, which was hugely important for our boutique brands, many of which are quite unique."—Robert Schaub, Marketing Program Manager, Best Western Hotels

An Elite Status for Video Production

97%  reduced time to create videos 480+ videos created in one year 50 Videos produced in a single week 94% cost reduction compared to agency prices

Currently, 24 employees within BWH Hotels use Lumen5, and between them they've created over 475 videos since the platform was first deployed just over a year ago. The company has also produced over 50 videos for social media, 25 of which were for a single hotel. 

Lumen5 has become a pivotal tool in the company, BWH Hotels has expanded the scope of its deployment well beyond its initial use case: videos for Google listings and TV displays around the hotel lobbies.  Since its deployment, Lumen5 has become a pivotal tool in the company. Best Western Hotels has expanded the video creation tool well beyond its initial use case: videos for Google listings and TV displays around the hotel lobbies.  Other use cases for Lumen5 range from: creating event videos for the company’s annual convention, consisting of seven district meetings and three regional meetings, which the platform has proved useful for both promotional videos and presentation videos. This past year, Best Western Hotels even used Lumen5 to create a presentation for its CEO.  To: the company also using Lumen5 for both internal and customer training videos, ranging from information on its Best Western Rewards loyalty program to details on procedures, strategies, and initiatives. Other video content includes tailored promotional videos for each hotel type, media for hotel lobby displays, and amenities highlights.  

Lumen5 cuts down on countless hours of work, and it's also fun to use. It doesn't feel like an addition to anyone's workload, and users are proud of what they're creating. That's the best part of it — it's such a time saver, and the AI component that draws out your storyboard for you is incredibly helpful. You don't have to think it through, you can just produce things faster and for a very cost-effective price. Robert Schaub Marketing Program Manager, Best Western Hotels

Video Example

Multiply your video content today..

• Senior Fellows
• Research Fellows
• Submission Guidelines
• Media Inquiries
• Commentary & Analysis
• Upcoming Events
• Past Events
• October 2021 War Studies Conference
• November 2020 War Studies Conference
• November 2018 War Studies Conference
• March 2018 War Studies Conference
• November 2016 War Studies Conference
• Class of 1974 MWI Podcast
• Urban Warfare Project Podcast
• Social Science of War
• Urban Warfare Project
• Project 6633
• Shield Notes
• Rethinking Civ-Mil
• Book Reviews

Select Page

John Spencer and Jayson Geroux | 06.28.21

The Battle of Stalingrad (modern-day Volgograd) occurred from August 23, 1942 to February 2, 1943 during World War II (1939–1945). The city is in the southwestern region of what was then the Soviet Union. The majority of the city rests on the west bank of the Volga River 970 kilometers southeast of Moscow. The Volga flows southwesterly into the city, passing through it before turning directly east and then curving gently to the southeast toward the Caspian Sea.

The battle was fought by the Axis powers of Army Group B—principally the German 6th Army commanded by Field Marshal Friedrich Paulus within the city—and the Soviet Union’s Stalingrad Front and its subordinate 62nd Army (commanded by General Vasily Chuikov) and 64th Army (commanded by General Mikhail Shumilov). Known as the biggest defeat in the history of the German Army, the battle destroyed Germany’s reputation of invincibility and sent the country into a more-or-less defensive mode for the duration of the war. The battle nearly guaranteed that Germany had begun the path to defeat on the Eastern Front.

The German Army Group South’s original strategic intent was to advance to and seize the Caucasus oil fields but German leader Adolf Hitler’s additional strategic desire to capture the city named after his rival, the Soviet Union’s Joseph Stalin, was too tempting to refuse. The city was also an attractive target because it produced Soviet armored fighting vehicles and other military equipment. Seizing the city would allow half of Army Group South to provide a certain level of protection to the other half by shielding the latter’s northeastern flank as it advanced to the Caucasus; thus, its capture at the operational level was considered crucial by the Germans. Hitler thus split Army Group South into two smaller army groups, with Army Group A continuing south toward the Caucasus while Army Group B diverted east toward the city.

In the opening months of the battle the German 6th Army drove hard for the city while the Italian 8th Army and Romanian 3rd Army guarded Army Group B’s northwestern flank along the meandering Don River and the German 4th Panzer Army and Romanian 4th Army guarded the southeastern flank along the salt lakes south of Stalingrad. Initial fighting for Stalingrad’s outskirts began on August 23, 1942 with an opening air offensive. The 6th Army’s Germans ground their way forward against the 62nd and 64th Armies’ Soviets, aided by the Luftwaffe and German artillery, which slathered the city with thousands of tons of high explosives and destroyed most of the city’s buildings. With an influx of refugees caused by the war, the city’s population was over nine hundred thousand by 1941 and although many had left a large number remained within the city, resulting in the deaths of an estimated forty thousand Russian civilians who were working in the military factories. The Soviets used the great amount of destruction to their advantage by adding man-made defenses such as barbed wire, minefields, trenches, and bunkers to the rubble, while large factories even housed tanks and large-caliber guns within.

The fighting in Stalingrad quickly turned into some of the most high-intensity urban combat in history. It was urban combat at its worst. The level of violence and resulting destruction between the two sides became truly unimaginable as the Germans pushed to capture the extreme northern and southern ends of the city by the end of August. German tactics followed a successful pattern: Luftwaffe airstrikes, then artillery, then advancing infantry with tanks in support. Unfortunately there were never enough artillery, infantry, and tanks to do the job swiftly and although the pattern was effective it still came with a high cost in casualties. Two large German offensives throughout September and October forced the Soviets to occupy only a nine-mile-long north-to-south strip that was only two to three miles wide along the west side of the Volga. The Soviets only held the narrow sliver of land by throwing consistent reinforcements to prevent them from reaching all of the Volga within the city limits. The length and width of the Volga itself prevented the Germans from encircling and isolating the city. This allowed the Soviets the ability to continually brave the repeated German Luftwaffe and artillery strikes as they ferried reinforcements across the river and into the urban fight. The Germans continued fighting into November with small raids and attacks that often degenerated into several days of sustained, lethal urban combat with little result but many casualties.

During the fighting, the Soviets recognized that the Italians and Romanians guarding the German flanks were a potential weakness and had throughout the autumn months increased the number of Soviet armies on both the northwestern and southeastern flanks to total over seven hundred thousand soldiers with 1,400 tanks. Along with the Stalingrad Front, the Southwestern and Don Fronts (each equivalent to an army group) launched Operation Uranus (November 19–23, 1942) with the intent of crushing the Italian, Romanian, and German armies around Stalingrad, linking up someplace west of the city and thus encircling the 6th Army within the city itself. On November 23, the two Soviet advances—the Southwestern and Don Fronts from the northwest and the Stalingrad Front from the southeast—met at the village of Kalach , just west of Stalingrad, and completed the task of encircling the Germans. With the 6th Army’s logistical ground support now unavailable, the Luftwaffe attempted to resupply Paulus’s troops by air for the next several weeks but the meager sixty tons of supplies a day was far short of the 550 tons needed daily by the 6th Army. The Soviets also limited the aerial resupply by advancing southwest away from the city to increase the distance between the German airfields and Stalingrad while also emplacing antiaircraft artillery guns to destroy attempted resupply runs.

While the Luftwaffe tried to resupply the 6th Army the Soviets now went on the offensive, the intensity of fighting within the city becoming more violent. Hundreds of “shock groups” consisting of fifty to a hundred soldiers broke into small groups to fight as highly lethal and lightly armed infantry-engineer squads of three to five personnel, moving swiftly and silently throughout the rubble or “hugging” the Germans near the frontlines to avoid the effectiveness of German airpower and artillery. Dozens of snipers took advantage of the destruction to find nearly perfect firing positions. Snipers successfully killed hundreds of German soldiers, providing a psychological boost to the Soviets while lowering German morale tremendously. Soviet tanks were used in a way that would unnerve armor crews of today: instead of being used for maneuver they were dug deep into the rubble, camouflaged, and used as pillboxes, their carefully prepared positions remaining unseen until they fired the first shot from just a short distance away. These tactics took a devastating toll on German personnel and vehicles over the following months.

Buildings and floors within Stalingrad changed hands dozens of times, and sometimes platoons and companies took several days and up to 90 and even 100 percent casualties just to win a building or a floor within it. Entire battles were fought over single buildings or complexes with names like the Martenovskii Shop, Pavlov’s House, the grain elevator, and the Commissar’s House. To add to the Germans’ misery, Russia suffered its worst winter in almost half a century with the temperature well below freezing on most days, conditions that became even worse at night or during severe winter storms. Thousands of Germans became medical casualties from both combat and the cold weather as a result, while the Soviets, acclimatized to such conditions, continued to grind away at German numbers with an almost unending violence.

Paulus, whose Army’s numbers throughout December 1942 and January 1943 were being reduced in vast quantities daily and with a majority of his soldiers on the brink of starvation, defied Hitler’s orders to fight to the bitter end; Hitler implied to Paulus, who had been promoted to the rank of field marshal during the battle, that he should commit suicide rather than capitulate. Paulus surrendered on February 2, 1943 . The casualties on both sides were horrendous—the result of two dictators feeding men, materiel, and machines into an urban fight all for the sake of taking or defending the city named after one of them. German casualties were estimated at four hundred thousand men with ninety-one thousand prisoners. Soviet casualties were estimated at over 750,000 . Historians have remarked that Stalingrad was the turning point of the fighting on the Eastern Front as it was the first public and large-scale loss suffered by the Germans in that theater of the war. It also demonstrated high-intensity urban combat at its most violent and brutal under the most challenging of weather conditions.

Lessons Learned

There are many lessons that can be taken from the battle. However, any appreciation of those lessons must be tempered by the caveat that some of them may not be applicable to present-day militaries. For example, Stalingrad took place in a theater with a large number of army groups with a total of a million soldiers involved on each side; modern armies are unlikely to fight with these numbers. Thus, the analysis on the lessons learned for Stalingrad will focus on those that can be applied to the present environment.

Strategically, Stalingrad illustrates that the reasons a nation would engage in high-intensity combat in dense urban terrain against a peer adversary may not be rational. The city was not a decisive piece of terrain for either side, but political reasons came to the fore: Hitler wanted to take the city named after his rival, Stalin; Stalin wanted to ensure that his namesake city did not fall. Arguably the battle was fought more for pride than for rational military or national objectives.

Operational reach is a function of intelligence, protection, sustainment, endurance, and combat power relative to enemy forces. The limit of a unit’s operational reach is its culminating point —the point at which a force no longer has the capability to continue its form of operations, offense or defense. The ability to resupply an army can become more important than tactical capabilities. In high-intensity urban operations a higher number of resources will be needed: four times the amount of ammunition required in non-urban environments and up to three times the amount of consumables such as rations and water are the norm. Operationally, then, Stalingrad emphasized the necessity of a military’s recognition of its own limits of operational reach and its culmination point. Once German forces could no longer be resupplied, they were defeated.

Due to the scale, duration, and intensity of the battle, Stalingrad offers enough tactical lessons to fill entire books. The importance of combined arms in urban operations was clearly one of the most important lessons demonstrated. After-action reports of the battle discussed how German armor was too vulnerable to enemy fire and worked better as fire support behind the infantry. Combined arms teams—armor, infantry, engineers, and fires from artillery and mortars—must be trained together to achieve the high level of cooperation, teamwork, and tactical capability required by high-intensity combat in dense urban terrain.

Heavily fortified urban infrastructure becomes critical in an urban defense and becomes major obstacles in an attack. At Stalingrad, entire battles fought over single buildings or complexes occurred frequently and lasted for hours, days, and sometimes weeks. Defenders must plan to use fortified buildings as strongpoints, while attackers must have plans to negotiate or reduce these structures. That does not necessarily mean just using heavy fires to eliminate strongpoints, although that is a consideration; the use of fires—aerial bombing, artillery, and mortars—are not the single solution. Stalingrad shows that fires alone do not eliminate defending enemies embedded deep in the urban terrain. The benefit of fires must also be weighed against how it will change the terrain for the attacker. Rubble limits maneuver and the effectiveness of critical capabilities like tanks.

The use of subterranean systems rises with the lethality of combat in urban terrain. Some soldiers described the conflict in Stalingrad as rattenkreig —“rat war”—because so much of it concentrated on controlling holes, cellars, and sewers throughout the city. Military forces must be prepared to use a large amount of resources—in particular manpower as armored fighting vehicles and large equipment cannot be taken into subterranean systems easily—and specific training must be conducted on how to fight in these tightly confined, dangerous spaces to boost soldiers’ confidence and proficiency.

Snipers are a force multiplier and as an enabler must be given to tactical maneuver units. Soviet snipers proved devastating to German forces due to their ability to hide in a seemingly endless number of locations and put effective fire on hundreds if not thousands of critical targets. Conversely, when dealing with a high sniper threat, military forces must have proactive contingency plans to effectively counter snipers. The use of counter-sniper teams; particular weapons systems such as a tanks, antitank platforms, and armored personnel carriers that are solely tasked to quickly engage and destroy enemy snipers; the use of smoke; and moving forces through the insides of buildings were among the solutions.

The overarching lesson for military forces is that adaptability and improvisation of existing systems becomes critical. For example, the Germans had several types of tracked antitank guns that were very useful in Stalingrad, where rubble and partially knocked-down walls provided them with cover up to their hulls. Deployed in hull defilade behind infantry the weapons proved highly effective.

The carnage, intensity, and scale of the Battle of Stalingrad made it one of the most memorable and referenced urban combat events in history. The battle has become largely synonymous with modern conceptions of high-intensity urban combat. Its lessons for today’s military forces are important, but they should be tempered with facts about what really happened as well as the vast amount of on-the-ground adaptations that were required by the two forces that fought in it.

John Spencer is chair of urban warfare studies at the Modern War Institute, codirector of MWI’s Urban Warfare Project, and host of the Urban Warfare Project Podcast . He previously served as a fellow with the chief of staff of the Army’s Strategic Studies Group. He served twenty-five years as an infantry soldier, which included two combat tours in Iraq.

Major Jayson Geroux is an infantry officer with The Royal Canadian Regiment and currently a member of the directing staff at the Canadian Armed Forces’ Combat Training Centre’s Tactics School. He has been involved in urban operations training for almost two decades and is the school’s urban operations subject matter expert and urban warfare historian, having participated in, planned, executed, and intensively instructed on urban operations for the past seven years. He has served twenty-six years in the Canadian Armed Forces, which included operational tours to the former Yugoslavia (Bosnia-Herzegovina) and Afghanistan.

A special thanks to Modern War Institute intern Harshana Ghoorhoo, whose initial research and framework of this and following case studies set the conditions for success.

The views expressed are those of the authors and do not reflect the official position of the United States Military Academy, Department of the Army, or Department of Defense, or that of any organization with which the authors are affiliated, including the Canadian Department of National Defence, the Canadian Armed Forces, and the Canadian Combat Training Centre and its Tactics School.

I recently saw the film Dredd on Netflix. The film was an interesting speculative scenario of urban and close quarters combat. The setting was an extreme built environment where the "judges" had to survive in a low intensity war situation and with non state actors.

The USSR had no initial plans to defend Stalingrad; it is not true that it was a prestige issue. Hitler divinding his curves, instead of concentrating them to take the Caucasus oilfields, however gave the Red Army the opportunity to defeat Paulus at Stalingrad and compel the evacuation of Army Group B, no matter how far out got towards Grozny. Which is what it did.

Forces, not curves.

Leave a reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 28 September 2024

Evaluation of five global AI models for predicting weather in Eastern Asia and Western Pacific

• Cheng-Chin Liu   ORCID: orcid.org/0009-0003-4112-072X 1 ,
• Kathryn Hsu   ORCID: orcid.org/0009-0003-2267-6802 1 ,
• Melinda S. Peng 2 ,
• Der-Song Chen 1 ,
• Pao-Liang Chang 1 ,
• Ling-Feng Hsiao 1 ,
• Chin-Tzu Fong 1 ,
• Jing-Shan Hong 1 ,
• Chia-Ping Cheng 1 ,
• Kuo-Chen Lu 1 ,
• Chia-Rong Chen 1 &
• Hung-Chi Kuo 3  

npj Climate and Atmospheric Science volume  7 , Article number:  221 ( 2024 ) Cite this article

Metrics details

• Environmental impact

Recent development of artificial intelligence (AI) technology has resulted in the fruition of machine learning-based weather prediction (MLWP) systems. Five prominent global MLWP model, Pangu-Weather, FourCastNet v2 (FCN2), GraphCast, FuXi, and FengWu, emerged. This study conducts a homogeneous comparison of these models utilizing identical initial conditions from ERA5. The performance is evaluated in the Eastern Asia and Western Pacific from June to November 2023. The evaluation comprises Root Mean Square Error and Anomaly Correlation Coefficients within the designated region, typhoon track and intensity predictions, and a case study for Typhoon Haikui. Results indicate that FengWu emerges as the best-performing model, followed by FuXi and GraphCast, with FCN2 and Pangu-Weather ranking lower. A multi-model ensemble, constructed by averaging predictions from the five models, demonstrates superior performance, rivaling that of FengWu. For the 11 typhoons in 2023, FengWu demonstrates the most accurate track prediction; however, it also has the largest intensity errors.

Similar content being viewed by others

Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciarán

A machine learning model that outperforms conventional global subseasonal forecast models

FuXi: a cascade machine learning forecasting system for 15-day global weather forecast

Introduction.

Turing’s 1 pioneer work “Computer Machinery and Intelligence” introduced The Turing Test, used by experts to measure computer intelligence even up to date. The term “artificial intelligence” (AI) was coined in 1956 at a Dartmouth summer workshop and machine learning (ML) was referred to by Arthur Samuel 2 . The advancements in AI/ML have experienced intermittent periods of stagnation over the past few decades. Significant progress in AI applications recently stems from vast datasets, rapid computational capabilities, and the availability of improved AI tools. Achievements of AI includes speech and vision recognizers, autonomous vehicles, cognitive computing, and expert systems, etc. These applications are increasingly penetrating various scientific and engineering communities. Many ML algorithms can be thought of as optimizing a nonlinear regression, with deep learning utilizing an extremely high-dimensional model. ML has proven to be an excellent tool for addressing complex, nonlinear, or stochastic challenges encountered in fields such as physics and Earth science.

Numerical weather prediction (NWP) models using nonlinear primitive equations with parameterizations accounting for sub-grid scale physical processes have been developed in the last 70 years. The success of NWP comes from supercomputing capabilities allowing high resolutions, improvements of physical parameterization, data assimilation strategy, and satellite retrievals. Recently, the Earth science community is adept at embracing AI/ML, as many AI concepts align closely with meteorology challenges, particularly data assimilation. Progress in AI meteorology has seen exponential growth since 2016, driven by achievements in ML techniques and sufficient computing resources, such as GPUs. There are many AI applications in Earth science such as PDE solving 3 , 4 and NWP post processing 5 , 6 , 7 .

Recently, there was a growing interest in utilizing AI/ML techniques to build ML-based (data-driven) weather and climate prediction (MLWP) models directly from atmospheric data. A variety proof of concept studies based on a few atmospheric variable comparisons has reveal that MLWP may have a great potential competing with traditional physical-based numerical weather prediction models 8 , 9 , 10 , 11 , 12 . However, the rather coarse spatial resolution data (~5 o ) used in these studies limited their forecast performances and applications. A chronicle review of the ML weather and climate modeling is given by de Burgh-Day and Leeuwenburg 13 .

Significant breakthrough emerged starting in 2022 for the development of global MLWP models applying to weather and climate predictions. FourCastNet system (FCN) 14 is the first MLWP model producing 0.25 o resolution forecasts using ECMWF ERA5 reanalysis 15 as its training data. The system applying Vision Transformer (ViT) 16 with the Fourier Neural Operator (FNO) 17 and Adaptive Fourier Neural Operators (AFNO) 18 for efficient computation in long-range dependencies in spatial-temporal data. A vision transformer is a deep learning model that breaks down an image into patches, processes them using transformers, and aggregates the information for object detection. FourCastNet was subsequently upgraded to version 2, FCN2, that uses Spherical Harmonics Neural Operators (SFNO) to build a more stable autoregressive model for weather prediction 19 . SwinRDM 20 is the first MLWP system to outperform ECMWF’s deterministic operational forecasting system, IFS-HRES, in 5-day forecasts at a spatial resolution of 0.25 o . Pangu-Weather 21 acquires promising medium-range performance that surpasses IFS-HRES with a multi-timescale model combination strategy based on 3D Earth-Specific Transformers (3DEST) in which the upper-air variables and the surface variables are embedded into a single deep network. The 3D data are propagated through an encoder–decoder architecture derived from the Swin transformer 22 , a variant of the vision transformer. GraphCast 23 takes the approach of applying the graph neural networks (GNN) 12 to a six-layer icosahedron grids with increasing resolution globally in which a set of objects and the connections between them are expressed as a graph. A 12-step autoregressive fine tuning is employed as the strategy for increasing the long-lead prediction accuracy. The model is more accurate in predicting 90% of the atmospheric variables compared with the IFS-HRES.

FuXi 24 model backbone is based on Swin transformer v2 25 with 3D cube embedding (called U-transformer) and is designed as a cascade of models (short for 0 ~ 5 days, medium for 5 ~ 10 days, and long for 10 ~ 15 days) optimized for different forecast lead times that is similar to the multiple time steps used in Pangu-Weather 21 . FengWu forecast system is constructed from a multi-modal and multi-task perspective in which each atmospheric state variable is treated as an individual modal and a cross-modal fuser transformer is applied to connect them 26 . To solve the long-lead prediction issue, the replay buffer mechanism is used, which is inspired by the reinforcement learning study 27 . The replay buffer stores the predicted results from previous optimization iterations and uses them as the current model’s input, which mimics the intermediate input error during the autoregressive inference stage.

Bouallegue et al. 28 conducted a comprehensive comparison between Pengu-Weather simulation and ECMWF operational IFS using the same initial conditions. The results are very promising for the MLWP model, with comparable accuracy from Pengu-Weather for both global metrics and extreme events, when verified against the IFS analysis and synoptic observations. Meanwhile, overly smooth forecasts, increasing bias with forecast lead time, and poor performance in predicting tropical cyclone intensity are identified as current drawbacks of ML-based forecasts. Charlton-Perez et al. 29 conducted a quantitative evaluation for FCN, Pangu-Weather, GraphCast, and FCN2 on the prediction of Storm Ciaran (2023), which has caused significant casualties and damage in Europe. The simulations of four MLWP models accurately capture the synoptic-scale structure of the winter cyclone including the position of the cloud head, shape of the warm sector and location of the warm conveyor belt jet. Meanwhile, all of the MLWP models underestimate the peak amplitude of winds associated with the storm. This appears to be the commonality of existing machine learning models with resolution limited by ERA5 as their training data.

The new developments on MLWP systems initiated a new era on weather predictions 30 . While all the aforementioned MLWP models have demonstrated performance compatible with or superior to conventional NWP models, and some comparisons between individual models have been analyzed, a homogeneous, comprehensive comparison among them is lacking. Additionally, the illustrations of tropical cyclone prediction skills, which serve as examples of extreme weather events, are selectively presented in their major publications.

An essential aspect of global weather prediction models is their role in generating initial and boundary conditions for regional weather forecast systems 31 . High resolutions in regional models facilitates representing of complex multi-scale processes crucial for high-impact weathers that directly affect human society directly. Biases and model errors present in the global forecast fields are thus inherited by regional models 32 . The influence of uncertainty in initial conditions usually diminishes over the course of the simulation length, whereas the impact of variations in lateral boundary forcing does not exhibit a clear trend.

As an operational weather prediction agency, the Central Weather Administration (CWA) is responsible for weather forecasts in Taiwan, encompassing a broad spectrum of temporal scales ranging from nowcasting to climate predictions. We utilize global forecast fields to drive our regional models, which feature high resolutions specifically tailored for the East Asia and western Pacific region 33 . The performance of the driving global forecast model significantly influences the accuracy of a limited area significantly 34 . Therefore, performance of the driving global model is critical for the success of our regional models. This study aims to contribute to this understanding.

Synoptic-scale predictions

In this section we present the performance of the five MLWP models and IFS in the area shown in Fig. 1 , covering the evaluation period from June to November 2023. Note that IFS is the operational NWP system at ECMWF and our MLWP simulations used ERA5 as the initial conditions, which is different from the IFS. We compute the standard metrics of latitude-weighted Root Mean Square Error (RMSE) and Anomaly Correlation Coefficient (ACC), commonly used in the meteorological community. ACC measures the spatial correlation between a forecast anomaly relative to climatology and is widely used for synoptic-scale prediction assessment.

The best tracks are categorized as straight (blue), curving (black), recurve (red), or irregular (green), with the names of the typhoons indicated beside their respective tracks. Track data interval is 6-h, and the dots shows 00 UTC of the day.

Figure 2 shows the RMSE and ACC for various variables, including the 500 hPa geopotential height, temperature at 850 hPa and at 2 m height, and zonal component of the wind at 10 m height. Based on RMSE, the best model is FengWu, followed by FuXi and GraphCast, with FCN2 trailing behind, and Pangu-Weather ranking last. IFS with a resolution of 9 km, lies in the middle of the group. Near the surface, there is a slight variation between FCN2 and FuXi. It is worth noting that ERA5 contains more observational data post-analysis data, while IFS uses the operational real-time analysis without the post-analysis data. This is reflected by the larger RMSE of IFS compared to the other five models at the initial time. With this in mind, we focus on the error trends when comparing IFS with the other models rather than the absolute magnitudes. In general, the slope of the RMSE from IFS aligns with the others, except for Pangu-Weather, which exhibits a larger upward trend. Regarding ACC (Fig. 2e–h ), the performance of the five MLWP models is similar to their RMSEs, with FengWu leading and Pangu-Weather trailing, and IFS falling in the middle of the group.

RMSE for a 500 hPa geopotential, b 850 hPa temperature, c temperature at 2 m height, and d zonal component wind at 10 m height of Pangu-Weather (red), FCN2 (green), GraphCast (orange), FuXi (light brown), FengWu (brown), the ensemble (light blue) of the five MLWP models, and IFS (blue), e – h ACC correspond to the same variables.

Weather forecasting inherently carries uncertainties due to the chaotic nature of weather systems and imperfect initial conditions. To address this uncertainty, ensemble forecasting is employed through perturbing the initial conditions and physical parameterizations 35 , 36 . There is also a multi-model ensemble approach that is widely recognized for its improved performance over individual models in various fields, including weather forecasting, climate modeling, and machine learning 37 . Hagedorn, et al. 38 provided a comprehensive analysis of why multi-model ensembles outperform individual models in seasonal forecasting. A multi-agency effort was established for the North American Multi-Model Ensemble (NMME) prediction system targeted at the seasonal forecasts that showed better performance compared to individual models 39 .

Due to their exceptional computational efficiency, MLWP models have been proposed as ideal for ensemble predictions from a single-model ensemble perspective. Here, we investigate the potential benefits of a multi-model ensemble approach by taking a simple multi-model ensemble strategy. We compute the RMSE and ACC of the ensemble by averaging forecast fields from all five MLWP models. Notably, when averaging the forecasts from the first three low-scoring models, the ensemble outperforms the individual models (figure not shown). As more models are added to the ensemble, the additional benefits become less pronounced. Ultimately, the performance of the ensemble comprising all five models approaches that of the best-performing model, which is FengWu in this study. The primary advantage of using a multi-model ensemble is the reduction in the error range. Specifically, for tropical cyclone track predictions, the ensemble consistently stays within the error range, preventing outliers from individual models. This point will be addressed further when evaluating typhoon predictions. Note that in this simple multi-model ensemble strategy, the uncertainty associated with initial conditions from different models is not accounted for. However, the ensemble strategy still demonstrates its conceptual advantage.

Another important metric for assessing model performance is the model bias, which represents the difference between the mean forecast state and the verification. In this study, we examine the position and strength of the western Pacific subtropical high (WPSH) system, a key feature of Hadley circulation’s substance. The variation of the WPSH is primarily influenced by central Pacific cooling/warming and positive atmosphere-ocean feedbacks between the WPSH and the Indo-Pacific warm pool oceans 40 . The position and strength of this system significantly influences various regional weather features such as monsoon circulations and TC movements. Interactions between a TC and the WPSH can lead to changes in cyclone trajectories, making them more prone to recurvature or maintaining a straight path based on the strength and position of the WPSH 41 .

We use the 500 hPa geopotential height contour as a measurement of the system, with the 5880 m line serving as a common reference in local operational communities and predicting of it from different models can be easily assessed. Figure 3 illustrates the average of the 168 h forecasted 5880 m contours in summer (JJA) and fall (SON) from each MLWP model, alongside the three-month mean from ERA5 serving as the verification. There is a seasonal migration of the WPSH westward from summer to fall. In both periods, the 5880 m contour line from ERA5 encompasses all lines from the MLWP models, indicating that all these models exhibit a weak bias for the WPSH. Among them, Pangu-Weather shows the largest weak bias, while FengWu exhibits the smallest bias, consistent with their RMSE and ACC scores. This observation holds significant importance in the maritime continent, where the Madden-Julian Oscillation (MJO) 42 , 43 is active, leading to large variabilities and far-reaching influences across the globe.

Contours show the period of ( a ) June, July, and August and ( b ) September, October, and November 2023 of Pangu-Weather (red), FCN2 (green), GraphCast (orange), FuXi (light brown), FengWu (brown), the ensemble (light blue) of those MLWP models mentioned above, and ERA5 (black).

Additionally, the 5760 m contour lines at higher latitudes from the five MLWP models are more closely clustered in both seasons. In general, tropical regions present greater challenges in weather prediction compared to mid-latitudes 44 , primarily due to more vigorous convective activities 45 . The simple analysis presented here suggests that MLWP models may also face greater challenges at lower latitudes, likely due to limitations inherited from their training data, which is the reanalysis relying on NWP model as its backbone.

Typhoon predictions

We assess the performance of the five MLWP models alongside IFS in predicting the 11 typhoons occurring between June and November 2023, excluding three short-lived ones (see best tracks in Fig. 1 ). The positions of the typhoons are identified by the storm center, determined as the minimum sea-level pressure in the forecast fields. Typhoon best track data provided by CWA was utilized for this analysis, and they are similar to those from the International Best Track Archive for Climate Stewardship (IBTrACS) database 46 , 47 . Track and intensity errors from the 7 models, including the five MLWP models, their ensemble, and IFS are depicted in Fig. 4 . Note that the averages are taken by the forecast lead time, which likely represent different stages of the life cycle for individual storms. In a case study presented later, we will illustrate how predictions for a storm can vary significantly in different stage.

a track errors (km), b absolute intensity errors (hPa). The total number of cases are listed at the bottom. Colors according to the models are the same as Fig. 2 . The filled squares show averaged track and intensity error every 24 h starting from the initial time.

Among the compared MLWP models, Pangu-Weather exhibited the largest track error, followed by FCN2, GraphCast, FuXi, and FengWu, progressively with smaller error up to 144 h. Subsequently, FengWu’s track error is higher than FuXi and slightly higher than GraphCast at 168 h. Overall, errors of IFS lie roughly in the middle of the group even though it has the smallest position error at the initial time (in the zoomed inlet). The ensemble performance was very close to FengWu. The benefit of using a multi-model ensemble is to reduce the range of error in tropical cyclone track prediction in which the ensemble is always within the range so that one would not get outliers (to be discussed more using Table 1 ). Comparing Fig. 4 with Fig. 2 indicates that, in general, track prediction performance aligns with the ranking by RMSE and ACC. Because TCs are steered, to first order, by the large-scale flow at mid-tropospheric levels 48 . Therefore, a weather prediction model with better ACC and RMSE scores usually also has better TC track prediction.

In terms of intensity prediction, the absolute errors are displayed to avoid cancellation of positive and negative relative errors, and all models exhibit weak biases. Although FengWu exhibited the smallest overall track errors, it had the largest intensity errors. Meanwhile, GraphCast and Pangu-Weather tied for the lowest intensity errors. It is noteworthy that IFS exhibited a much lower initial intensity bias compared to the five MLWP models, which used ERA5 as their initial conditions. Nevertheless, the trend of intensity errors was similar across all models. When comparing Pangu-Weather’s performance with IFS on TC intensity prediction, Boualleguea et al. 28 also demonstrated that Pangu-Weather performed poorly than IFS. Their Fig. 8b suggests that the superiority of IFS may stem from better initial (higher) intensity due to its higher resolution. However, the slope of the error does not necessarily indicate that IFS predicts intensity changes better than Pangu-Weather. An interesting observation is that all models exhibited a decreasing intensity error from 120 to 168 h. This is because TCs near the end of their life cycle are usually weak and the ranges of predicted intensities are smaller.

We conducted two sets of statistical significance tests for typhoon track and intensity predictions: one comparing Pangu-Weather (PW) with the other models, and one comparing FengWu (FW) with the other models. Since the typhoon track and intensity errors do not follow a normal distribution, we adopted the Mann-Whitney U test 49 , 50 to assess statistical significance. The results indicate a 95% confidence level in the comparison of track errors between the selected model (PW or FW) and other MLWP models. Additionally, there is a 90% confidence level in the difference in TC intensity errors between FW and the other MLWP models.

It is widely acknowledged that TC track predictions can exhibit significant diversification for individual typhoons. Instead of displaying predicted tracks for all typhoons from all models, Table 1 offers further statistical insights into our evaluations. We subjectively classify track characteristics into 4 categories: straight, curving, recurve, and irregular, as illustrated in Fig. 1 . Evaluation is conducted at 96 h due to its substantially higher number of verifications available, as the number of cases decreases significantly with longer lead times. The best-performing model for track prediction is FengWu, with one typhoon prediction leading in all 4 categories, consistent with the averaged track errors shown in Fig. 4 . Interestingly, despite Pangu-Weather exhibiting the largest average track error, it performs the best for three individual typhoons. Additionally, IFS achieves the highest scores for two typhoons in the recurving category, while GraphCast and FuXi each lead in one storm.

While TC intensity prediction may not be the primary focus of a global model, it is still pertinent to discuss the performance of the five models. FengWu, despite excelling in track prediction, exhibits a reversal in the performance for intensity prediction, failing to rank as first for any individual typhoon—a result consistent with its largest absolute average errors depicted in Fig. 4b . Conversely, Pangu-Weather leads in intensity prediction for 4 typhoons, while both FuXi and GraphCast lead for 3 typhoons each. It’s worth noting that the numbers shown in column 7 of Table 1 for the best-performing model are significantly smaller than the averages displayed in Fig. 4b , which range from 25 to 35 hPa. These more detailed performance assessments on track and intensity predictions from different models further underscore the diversification of TC predictions. The IFS is not included in the intensity evaluation (columns 7 and 8) due to its advantage of low intensity bias at the initial time, attributed to its high resolution.

Case Study-Typhoon Haikui (2023)

Typhoon Haikui (blue track in Fig. 1 ) was the first major storm to hit Taiwan since Bailu in 2019. Behind Typhoon Saola (left green track in Fig. 1 ), Haikui began its life as a broad low-pressure system on August 27 near the Northern Mariana Islands. The system intensified to a tropical storm the next day and was named Haikui by the Japan Meteorological Agency (JMA). In the subsequent days, Haikui reached a tropical storm strength and eventually became a typhoon, before making landfall near Taitung City, Taiwan, on September 3. Haikui also enhanced the southwest monsoon in the Philippines, causing extensive rainfall in Luzon. As it stalled over Pearl River Delta in China, the remnants of Haikui induced torrential rain in Hong Kong resulting in the issuance of a Black Rainstorm Signal for 16 h, the longest duration ever since the rainstorm warning system was implemented in 1992. Overall, Haikui caused US$2.31 billion worth of damage during its onslaught.

As listed in Table 1 , the prediction of the track for Typhoon Haikui was the most inaccurate (average and individual errors) among all typhoons in the western North Pacific in 2023. We present the track predictions from 12 UTC on August 28 to 12 UTC on September 4 for Typhoon Haikui from individual models, including IFS and the five MLWP models (Fig. 5 ). Among them, IFS exhibited the largest average track error, exceeding 1000 km at 96 h with a significant poleward bias, mainly from the early stage (Fig. 5f ). Both GraphCast and FCN2 (Fig. 5b, c ) also showed substantial track errors for the first two watches, with their predicted tracks resembling that of IFS. Meanwhile, FengWu’s predicted tracks were closely aligned with the best track (Fig. 5e ), with the averaged track error of only 41 km at 96 h, making it the best-performing model for Haikui’s track prediction. The second best is Fuxi (Fig. 5d ), followed by Pengu-Weather (Fig. 5a ).

The black line with typhoon marks represents the best track, while lines with other colors and marks indicate forecasts at different initial times of ( a ) Pengu-Weather, ( b ) GraphCast, ( c ) FCN2, ( d ) FuXi, ( e ) FengWu, and ( f ) IFS. The open typhoon mark shows the best track location at 12 UTC 28 August while closed typhoon marks show the best track locations at 00 UTC each day with date beside. Marks in other colors indicate the forecast initial time.

Next, we examine the potential link between the predicted tracks and the position of the WPSH. For consistency and ease of comparison among model predictions, the WPSH system is represented by the 5880 m height at 500 hPa, as referenced for its seasonal verification (Fig. 3 ). Figure 6 illustrates two 96 h forecasts, starting at 00 UTC 29 August and then 6 h later at 06 UTC on 29 August, from IFS and the five MLWP models. For the first forecast, all models exhibit a poleward bias in their predictions, with the IFS showing the largest and FengWu and Fuxi the smallest (Fig. 6a ). While only the 96-h geopotential pattern are displayed (Fig. 6b ), the distribution of the western edges of the WPSH generally aligns with their individual tracks. In the second example, starting 6 h later, the WPSH predicated by the five MLWP models have all migrated westward, while in IFS, the system lagged and remained close to its position predicted 6 h earlier (blue lines in Fig. 6b, d ). Following the movement of the WPSH, all five MLWP models adjusted their predicted tracks to be mainly westward, in line with the best track, while the IFS maintained its northwestward track (Fig. 5f ). The predicated tracks by IFS are very similar in these two forecasts, showing a significant poleward bias (blue lines in Fig. 6a, c ), which contributed to its overall large track error for Haikui (Table 1 ).

The 96 h best track and predicted tracks are starting from 00 UTC 29 September ( a ) and from 12 UTC 29 September ( c ). b and d show the corresponding geopotential height contour line at 96 h forecast of five MLWP models, their ensemble, IFS, and the analysis of ERA5 at the same time. Colors according to the models are the same as Fig. 2 and the black line in ( a ) and ( c ) show the best track, while it in ( b ) and ( d ) show the geopotential height of ERA5. The open diamonds and the filled squares on ( a ) and ( c ) show Haikui’s location every 24 h starting from the initial time.

Tropical cyclone movement can be influenced by many surrounding synoptic-scale and mesoscale features 51 , 52 . The analysis presented here only provide the first-order influence by the WPSH, as commonly assessed by operational weather prediction centers in eastern Asia and western Pacific region. As shown in Fig. 5f , the IFS eventually adjusted its predicted track to be mainly westward after the first four bad forecasts. Further in-depth analysis is required to fully understand the dynamics behind it.

High-resolution data and model configuration are critical for representing and simulating complex mesoscale phenomena. Thus, we rely on regional models for local area weather predictions, and for extreme weather systems such as typhoons. These models can offer significant benefits where local weather is influenced by factors like islands, coastlines, topography, and land/sea contrast. While MLWP models have demonstrated impressive performance in synoptic-scale systems, their effectiveness can vary for systems involving multiple scales (such as typhoons) and remain to be thoroughly examined. The TWRF (Typhoon WRF) system is a regional NWP system developed by CWA based on the ARW WRF model 53 , 54 , dedicated for typhoon prediction in East Asia with a focus on Taiwan. TWRF is two-way nested regional model with 15 km and 3 km resolutions in the outer and inner domain. Previous studies have verified its superior performance 33 , 34 , and its predictions are currently displayed on the NOAA hurricane analysis and forecast system website. In the following analysis of Haikui during its passage over Taiwan, we also include the prediction from TWRF (15 km version) for comparison.

Rainfall induced by a TC on Taiwan is intricately linked to its track, which determines how the typhoon interacts with the island’s complex terrain 55 . The case study starts on 12 UTC 2 September and the two-day accumulated rainfalls ending on 12 UTC 4 September will be evaluated. Here, we focus on the rainfall prediction from two MLWP models, GraphCast and FuXi, both of which include precipitation forecasts in their outputs. Also included in the comparison are two NWP models, IFS and TWRF (15 km resolution), along with the precipitation data from ERA5. First, we discuss the predicted tracks and intensities for the two-day period (Fig. 7 ). The predicted tracks from all models moved westward, hitting Taiwan on the second day with a small diversion after passing over the island. Notably, the IFS (blue line) has the best track prediction for this period, having corrected its significant poleward bias observed in the early stage (Fig. 5f ). All five models place their cyclone centers near the west coast of the island, to the west of the Central Mountain, at the 24 h forecast. By 48 h, the centers are positioned in the middle of the Taiwan Strait.

a best, analysis, and predicated tracks for two days starting on 12 UTC 2 September 2023 on map with terrain height (m). The open diamonds and the filled squares on ( a ) show Haikui’s location every 24 h starting from the initial time. b shows the 10-m maximum wind speed (m s -1 ) and c shows the minimum sea-level pressure (hPa) corresponding to ( a ). The open diamonds and the filled squares on ( b ) and ( c ) show Haikui’s intensity every 6 h starting from the initial time. Colors according to the models are the same as Fig. 2 with ERA5 in color gray.

Figure 7b, c illustrate the predicted maximum wind speed and minimum sea-level pressure associated with Haikui over the 48 h period starting at 12 UTC 2 September. Haikui weakened significantly after passing over Taiwan, with the maximum wind decreasing from 40 to 25 m s - 1 and the central pressure increasing from 950 hPa to 987 hPa, according to the best track data. Among the model intensity forecasts, IFS and TWRF are similar, both showing higher intensity and closely matching the best track due to their higher resolutions. The intensity evolution predicted by GraphCast nearly aligns with that from ERA5. Meanwhile, Fuxi predicted the weakest typhoon for the first 24 h. There is a strong relationship between the wind and pressure profile for this case, which is also observed in other typhoon cases we examined and at longer lead time (figures not shown). In investigating the prediction of a mid-latitude winter storm, Charlton-Perez et al. 29 did not find as strong a relationship between wind and pressure variations from the MLWP models they evaluated.

The capability of MLWP models to predict TC formation is also examined using Haikui as an example. The tropical disturbance that can be traced as precursor of Haikui developed into a tropical storm at 00 UTC on 28 August 2023. This time is referred to as the TC formation time in our analysis, Table 2 lists the dates in the predictions of each model in which a disturbance as the precursor of Haikui can be identified, following the criteria established by Tsai et al. 56 . Among them, FCN2 was the earliest to predict Haikui’s formation with the lead time of six days, while the others demonstrated a predicative ability to forecast formation within four to five days. Predicting tropical cyclone genesis usually involves more longer time scales and is best handled by ensemble system 57 . The single case presented here only provides a glimpse of the potential capability of MLWP models. We devote more in-depth research to this topic in another study.

Figure 8 illustrates the two-day accumulated rainfall from FuXi, GraphCast, ERA5, IFS, and TWRF (15 km). The last panel (Fig. 8h ) displays the accumulated rainfall retrieved by the Quantitative Precipitation Estimation and Segregation Using Multiple Sensor (QPESUMS) algorithm from CWA 58 , used for verification. Taiwan island is predominantly covered by the Central Mountain range with e peak about 4000 meters (Fig. 7a ). As Haikui approached Taiwan from the east and passed its southern part, the rainfalls are accumulated mostly on the upwind side of the Central Mountain ridge, which in this case on the eastern side.

Predicted rainfalls (mm) of ( a ) FuXi, ( b ) GraphCast, ( c ) ERA5, ( d ) IFS, ( e ) TWRF (15 km), and estimated rainfalls (mm) of ( f ) QPESUMS.

The maximum accumulated rainfall observed in the QPESUMS verification exceeds 700 mm over two days (Fig. 8h ). Among all five models examined here, only TWRF reached this extreme value (Fig. 8e ). The operational model IFS produced very good result in terms of distribution (Fig. 8d ), with its maximum reaching 600 mm. The ERA5 (Fig. 8c ) shows a rainfall pattern similar to IFS but with less details structure and a weaker peak intensity, in the range of 300–400 mm. For comparison, the IFS and ERA5 have resolutions of 0.25 o and TWRF presented here has a 15 km resolution.

The only two precipitation outputs available from the five MLWP models are Fuxi and GraphCast (Fig. 8a, b ). While the general patterns of the precipitations align with the verification, both AI model-generated rainfall amounts are smaller than those of the verification and are also less than those of the two dynamic NWP models (IFS and TWRF). Between them, GraphCast produced larger amount, reaching 200 mm, while Fuxi recorded rainfall in the range of 130–150 mm. Despite a small track difference, GraphCast predicted higher intensity than Fuxi during the first 24 h (Fig. 7b, c ). This intensity bias in Fuxi contributes to its smaller accumulated rainfall. Additionally, the circular rainfall pattern over Taiwan Strait between Taiwan Island and south eastern China, is also much weaker in Fuxi compared to other models. These intensity and precipitation bias may stem from Fuxi’s longer time step, which prioritizes longer lead time forecasts.

In investigating the capability of AI models in predicting an extreme weather event of Storm Ciaran (2023), Charlton-Perez et al. 29 noted that the four AI models they examined (FCN, Pangu-Weather, GraphCast, and FCN2) failed to accurately capture the structure and magnitude of the winds as revealed in ERA5. Their study suggested that the weaker winds predicted by these ML models, compared to NWP model forecasts, are not merely a result of being trained on a coarse resolution dataset. AI models often produce smoother results due to several factors: regularization techniques 59 , data averaging 60 , optimization objectives 37 , and noise reduction 61 . The analysis presented in this study and by Charlton-Perez et al. 29 may offer AI modelers insights for future improvements of their models.

In the rapidly advancing field of MLWP models, five standout global weather prediction systems have emerged: Pangu-Weather, FCN2, GraphCast, FuXi, and FengWu. Despite employing distinct AI/ML technologies, all five systems use ERA5 for training, spanning approximately 39 years. Publications on these models have demonstrated comparable, and in some cases, superior performance compared to the ECMWF’s traditional NWP model, IFS, while exhibiting several orders of magnitude greater computational efficiencies.

Given the intricacies of local geographic profile in regional areas, high resolution models are essential for accurately simulating multi-scale processes responsible for high-impact weather phenomena. The efficacy of regional models, however, depends critically on the initial and boundary conditions provided by global models 34 , 62 , 63 . The recent emergency of MLWP opened a new avenue for the adoption of these models in operational forecasting. Therefore, evaluating the performance of global MLWP models within a limited area is warranted.

This study independently evaluates the performance of the five aforementioned MLWP models in East Asia and Western Pacific over a six-month period from June to November in 2023. We conducted our simulations using the codes provided by the respective model developers, which are available on their websites. The initial conditions for our simulations are sources from sourced from ERA5, remaining identical for all models, with a forecast integration period set at 168 h. Additionally, we include comparisons with the predictions generated by IFS, noting that IFS has a higher resolution of 9 km compared to 0.25 o resolution of the MLWP models inherited from ERA5. Using ERA5 for initial conditions is not feasible for operational purposes, thereby potentially altering the performance comparisons when transitioning to an operational environment. Nevertheless, our primary focus remains on a consistent comparison of MLWP models, aiming to mitigate uncertainties stemming from varied initial conditions. In future studies, we plan to utilize other reanalysis, such as those from NCEP, to further enhance our insights in this endeavor.

We computed the latitude-weighted RMSE and ACC against valid ERA5 reanalysis for the evaluation in the East Asia and western Pacific region, which is of significant interest for our operational weather prediction. The region is also of great interest to meteorological community due its vigorous phenomena across a broad spectrum, including tropical cyclone activities, monsoon circulations, and the Madden-Julian Oscillation (MJO). The ACC and RMSE scores indicate that the best-performing model is FengWu, followed by FuXi and GraphCast, then FCN2, with Pangu-Weather ranking last. Notably, the IFS, with a resolution of 9 km, places it in the middle of the group. The trends and ranks of RMSE and ACC are consistent. Additionally, we conducted a simple ensemble average of the five MLWP models. The performance of the ensemble is comparable to FengWu.

We evaluated the prediction skill for TCs as an extreme event example. For the 11 typhoons (excluding three short-lived ones) that occurred in the western North Pacific in 2023, FengWu demonstrated the best track prediction among all models and led in four individual typhoons. The performance of the track prediction generally aligned with the ACC and RMSE scores. While Pangu-Weather has the largest averaged track error, it still performed best for three individual typhoons. The IFS excelled in the intensity prediction, attributed to its higher resolution and minimal initial intensity bias. Surprisingly, FengWu exhibited the poorest intensity prediction skill. A more detailed examination of typhoon prediction for individual cyclones highlights diversification in model performance, indicating potential for future improvements.

The seasonal average of the 7-day prediction of the WPSH for the five MLWP models all indicates weak biases. A case study of Typhoon Haikui demonstrated close relationship between the predicted track and the position of the WPSH. Haikui was a mostly westward-moving typhoon that made landfall in Taiwan, causing significant rainfall in countries it encountered. The challenge presented in the early stage of Haikui led to largest track errors for some models, with IFS showing the largest among them. While these models can closely approximate the typhoon’s track and some models generate reasonably accurate rainfall patterns, their resolution may not suffice for regional applications. Consequently, there remains a necessity for high-resolution regional models to offer detailed meteorological information, encompassing not only rainfall predictions but also variables such as temperature extremes. It is evident that MLWP models are progressing towards higher resolution predictions, whether through enhanced training techniques or downscaling predictions 64 , 65 . Another consideration for future development is the adoption of additional data within the existing reanalysis data. Further rigorous verifications of these models are still required.

The pace of development in MLWP is truly remarkable, with new systems emerging rapidly. Oskarsson et al. 66 have notably introduced a regional model based on GraphCast for regional weather forecasting around the Nordic area. However, a significant challenge in daily short-term prediction arises from the lack of high-resolution regional reanalysis data for training purposes. To address the challenge of insufficient resolution in ERA5 for TC intensity prediction, the FengWu group has developed the Multi-modal multi-Scale Causal AutoRegressive model (MSCAR) 67 . This innovative approach combines satellite images for TCs with ERA5 reanalysis data allowing for the extraction of causal relationships across these multi-modal datasets to enable global TC intensity autoregressive forecasting. The results of MSCAR show promising short-term performance, indicating a new pathway for the development of high-resolution AI/ML models in the fields of extreme weather predictions.

In October 2023, ECMWF launched its own AI prediction system 68 , AIFS (where “I” denotes both AI and IFS), marking a significant advancement in the field. AIFS utilizes Graph Neural Networks technology and shares the same grid structure as IFS. The current iteration of AIFS boasts 13 vertical levels and a 0.25 o resolution 69 , with preliminary results indicating its superior performance compared to IFS.

As with conventional NWP models, MLWP models are continually evolving, with efforts focused on extending forecast lead times into sub-seasonal, to seasonal prediction, and ultimately climate predictions. A notable advantage of the MLWP models is their ability to generate large numbers of ensemble members by perturbing the initial fields in the current state, all while benefiting from their extremely high computational efficiency. We also advocate for the utilization of multi-model ensembles in addition to single-model ensembles, with direct applications including the use of multi-model ensemble predictions as boundary conditions for high-resolution regional models. Given that MLWP systems heavily rely on data, collaboration between the AI and meteorological communities is essential to enhance the prediction of extreme weather events by leveraging high-resolution and reliable localized data.

In this study, we conduct an independent evaluation of the following prominent MLWP models: Pangu-Weather, FCN2, GraphCast, FuXi, and FengWu (Table 3 ), all with horizontal resolution of 0.25 o . While some models have multiple versions with different numbers of vertical levels, we use a 13-level configuration and 6 h time step for all models. We conducted our own simulations for each model using the codes published by model developers, available on arXiv ( http://arxiv.org ). ERA5 reanalysis are used as the initial condition in our simulations, which is the training data for all five MLWP models. The ML algorithm (backbone) used in these five MLWP models are briefly summarized in Table 3 .

We compare independently performances of the five global MLWP systems in a region covering East Asia and western North Pacific. This is the area for which CWA is responsible for daily operation of weather predictions. The area also covers the region of most rigorous tropical cyclone (TC) or typhoon activities. Our evaluation period spans from June to November 2023, during which predictions of 11 typhoon cases will be assessed, with three short-lived ones excluded from the analysis. We conduct simulations of each model with identical initial conditions from ERA5 and the forecasts are also verified against ERA5 reanalysis at the valid time. The forecast lead time is 168 h (7 days) for the simulation.

The evaluation matrix includes RMSE, ACC of representative atmospheric variables, and predicted track and intensity errors. The ERA5 reanalysis are used as verification for the forecast fields.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Turing, A. M. Computing machinery and intelligence. Mind 49 , 433–460 (1950).

Article   Google Scholar  

Samuel, A. L. Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 3 , 210–229 (1959).

Bar-Sinai, Y., Hoyer, S., Hickey, J. & Brenner, M. P. Learning data-driven discretizations for partial differen tial equations. Proc. Natl. Acad. Sci. 116 , 15344–15349 (2019).

Article   CAS   Google Scholar  

Kochkov, D. et al. Machine learning–accelerated computational fluid dynamics. Proc. Natl. Acad. Sci. 118 , e2101784118 (2021).

Gröenquist, P. et al. Deep learning for post-processing ensemble weather forecasts. Philos. Trans. Roy. Soc. A379 , 20200092 (2021).

Hess, P. & Boers, N. Deep learning for improving numerical weather prediction of heavy rainfall. J. Adv. Model. Earth Syst. 14 , e2021MS002765, https://doi.org/10.1029/2021MS002765 (2022).

Rojas-Campos, A. et al. Postprocessing of NWP precipitation forecasts using deep learning. Wea. Forecast. 38 , 487–497 (2023).

Dueben, P. D. & Bauer, P. Challenges and design choices for global weather and climate models based on machine learning. Geosci. Model Dev. 11 , 3999–4009 (2018).

Weyn, J. A., Durran, D. R. & Caruana, R. Can machines learn to predict weather? Using deep learning to predict gridded 500‐hPa geopotential height from historical weather data. J. Adv. Model. Earth Syst. 11 , 2680–2693 (2019).

Weyn, J. A., Durran, D. R. & Caruana, R. Improving data‐driven global weather prediction using deep convolutional neural networks on a cubed sphere. J. Adv. Model. Earth Syst. 12 , e2020MS002109 (2020).

Rasp, S. & Thuerey, N. Data-driven medium-range weather prediction with a Resnet pretrained on climate simulations: A new model for weather bench. J. Adv. Model. Earth Syst. 13 , 2020MS002405 (2021).

Keisler, R. Forecasting global weather with graph neural networks. arXiv , 2202.07575v1, https://doi.org/10.48550/arXiv.2202.07575 (2022).

de Burgh-Day, C. O. & Leeuwenburg, T. Machine learning for numerical weather and climate modeling: A review. Geosci. Model Dev. 16 , 6433–6477 (2023).

Pathak, J. et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators, arXiv , 2202.11214v1, https://doi.org/10.48550/arXiv.2202.11214 (2022).

Hersbach, H. et al. The ERA5 global reanalysis. Q. J. R. Meteorol. Soc. 146 , 1999–2049 (2020).

Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv , 2010.11929v2, https://doi.org/10.48550/arXiv.2010.11929 (2020).

Li, Z., et al & Anandkumar Fourier neural operator for parametric partial differential equations. https://doi.org/10.48550/arXiv.2010.08895 (2020).

J. Guibas, M. Mardani, et al. Adaptive Fourier Neural Operators: Efficient token mixers for transformers. arXiv , 2111.13587v2, https://doi.org/10.48550/arXiv.2111.13587 (2021).

Bonev, B., et al. Spherical Fourier Neural Operators: Learning stable dynamics on the sphere. arXiv , 2306.03838v1, https://doi.org/10.48550/arXiv.2306.03838 (2023).

Chen, L., Du, F., Hu, Y., Wang, Z. & Wang, F. SwinRDM: integrate swinRNN with diffusion model towards high-resolution and high-quality weather forecasting. arXiv , 2306.03110v1, https://doi.org/10.48550/arXiv.2306.03110 (2023).

Bi, K., et al. Pangu-Weather: A 3D high-resolution model for fast and accurate global weather forecast, arXiv , 2211.02556v1, https://doi.org/10.48550/arXiv.2211.02556 (2022).

Liu, Z., et al. Swin transformer: hierarchical vision transformer using shifted windows. Proc. IEEE/CVF Int. Conf. on Computer Vision (ICCV), virtual 10012–10022 (2021).

Lam, R. et al. Learning skillful medium-range global weather forecasting. Science 382 , 1416–1421 (2023).

Chen, L. et al. FuXi: a cascade machine learning forecasting system for 15-day global weather forecast. npj Clim. Atmos. Sci. 6 , 190 (2023).

Liu, Z. et al. Swin transformer v2: Scaling up capacity and resolution. Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , New Orleans, LA, pp. 11999–12009 (2022).

Chen, K. et al. FengWu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv , 2304.02948v1, https://doi.org/10.48550/arXiv.2304.02948 (2023).

Schaul, T., Quan, J., Antonoglou, I. & Silver, D. Prioritized experience replay. arXiv ,1511.05952v4, https://doi.org/10.48550/arXiv.1511.05952 (2015).

Bouallegue, Z. B. et al. The Rise of Data-Driven Weather Forecasting A First Statistical Assessment of Machine Learning–Based Weather Forecasts in an Operational-Like Context. BAMS 105 , E864–E883 (2024).

Charlton-Perez, A. J. et al. Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciarán. npj Clim. Atmos. Sci. 7 , 93 (2024).

Schultz, M. G. et al. Can deep learning beat numerical weather prediction? Philos. T. Roy. Soc. A. 379 , 20200097 (2021).

Mass, C. F. & Kuo, Y.-H. Regional real-time numerical weather prediction: Current status and future potential. Bull. Am. Meteor. Soc. 79 , 253–263 (1998).

Wu, W., Lynch, A. H. & Rivers, A. Estimating the uncertainty in a regional climate model related to initial and lateral boundary conditions. J. Clim. 18 , 917–933 (2005).

Hsiao, L.-F., Chen, D.-S., Hong, J.-S., Yeh, T.-C. & Fong, C.-T. Improvement of the numerical tropical cyclone prediction system at the Central Weather Bureau of Taiwan: TWRF (Typhoon WRF). Atmosphere 11 , 657 (2020).

Hsiao, L.-F. et al. Blending of global and regional analyses with a spatial filter: Application to typhoon prediction over the western north Pacific Ocean. Wea. Forecast. 30 , 754–770 (2015).

Leutbecher, M. & Palmer, T. N. Ensemble forecasting. J. Comput. Phys. 227 , 3515–3539 (2008).

Magnusson, L. et al. ECMWF activities for improved hurricane forecasts. Bull. Am. Meteorol. Soc. 100 , 445–458 (2019).

Bishop, C. M. Pattern Recognition and Machine Learning. Springer New York 2006 (2006).

Hagedorn, R., Doblas-Reyes, F. J. & Palmer, T. N. The rationale behind the success of multi-model ensembles in seasonal forecasting – I. Basic concept. Tellus 57A , 219–233 (2005).

Google Scholar  

Kirtman, B. et al. The North American multimodel ensemble: Phase-1 seasonal-to-interannual prediction; Phase-2 toward developing intraseasonal prediction. Bull. Am. Meterol. Soc. 95 , 585–601 (2014).

Wang, B., Xiang, B. & Lee, J.-Y. Subtropical high predictability establishes a promising way for monsoon and tropical storm predictions. Proc. Natl. Acad. Sci. USA 110 , 2718–2722 (2013).

George, J. E. & Gray, W. M. Tropical cyclone recurvature and noncurvature as related to surrounding wind-height fields. J. Appl. Meteor. 16 , 34–42 (1977).

Madden, R. A. & Julian, P. R. Description of global-scale circulation cells in the tropics with a 40–50 day period. J. Atmos. Sci. 29 , 1109–1123 (1972).

Madden, R. A. & Julian, P. R. Observations of the 40–50-day tropical oscillation - a review. Mon. Wea. Rev. 122 , 814–837 (1994).

Sobel, A. H. Tropical weather. Nat. Educ. Knowl. 3 , 2 (2012).

Mapes, B. E. & Houze, R. Cloud clusters and superclusters over the oceanic warm pool. Mon. Wea. Rev. 121 , 1398–1415 (1993).

Knapp, K. R., Kruk, M. C., Levinson, D. H., Diamond, H. J. & Neumann, C. J. The International Best Track Archive for Climate Stewardship (IBTrACS): Unifying tropical cyclone data. Bull. Am. Meterol. Soc. 91 , 363–376 (2010).

Knapp, K. R., Diamond, H. J., Kossin, J. P., Kruk, M. C. & Schreck, C. J. I. International Best Track Archive for Climate Stewardship (IBTrACS) Project, Version 4. NOAA National Centers for Environmental Information, https://doi.org/10.25921/82ty-9e16 [access date: July 2023] (2018).

Velden, C. & Leslie, L. The basic relationship between tropical cyclone intensity and the depth of the environmental steering layer in the Australian region. Wea. Forecast. 6 , 244–253 (1991).

Mann, H. B. & Whitney, D. R. On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18 , 50–60 (1947).

Wilcoxon, F. Individual comparisons by ranking methods. Biometrics Bull. 1 , 80–83 (1945).

Holland, G. J. The Global Guide to Tropical Cyclone Forecasting . Geneva: World Meteorological Organization. (Ed.) (1993).

Emanuel, K. A. Tropical cyclones. Annu. Rev. Earth Planet. Sci. 31 , 75–104 (2003).

Hsiao, L.-F., Liou, C.-S., Yeh, T.-C. & Guo, Y.-R. A vortex relocation scheme for tropical cyclone initialization in Advanced Research WRF. Mon. Wea. Rev. 138 , 3298–3315 (2010).

Hsiao, L.-F. et al. Application of WRF 3DVAR to operational typhoon prediction in Taiwan: Impact of outer loop and partial cycling approaches. Wea. Forecast. 27 , 1249–1263 (2012).

Lee, C.-S., Huang, L.-R., Shen, H.-S. & Wang, S.-T. A climatology model for forecasting typhoon rainfall in Taiwan. Nat. Hazards 37 , 87–105 (2006).

Tsai, H., Lu, K., Elsberry, R. L., Lu, M. & Sui, C. Tropical Cyclone–like Vortices Detection in the NCEP 16-Day Ensemble System over the Western North Pacific in 2008: Application and Forecast Evaluation. Wea. Forecast. 26 , 77–93 (2011).

Elsberry, R. L., Tsai, H.-C., Chin, W.-C. & Marchok, T. P. ECMWF Ensemble Forecasts of Six Tropical Cyclones That Formed during a Long-Lasting Rossby Wave Breaking Event in the Western North Pacific. Atmosphere 2024, 15 , 610 (2024).

Chang, P.-L. et al. An Operational Multi-Radar Multi-Sensor QPE System in Taiwan. Bull. Amer. Meteor. Soc . 102 , E555–E577 (2021).

Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning. MIT Press , Chapter 7 (2016).

Breiman, L. Bagging predictors. Mach. Learn. 24 , 123–140 (1996).

Ng, A. Y. Feature selection, L1 vs. L2 regularization, and rotational invariance. In Proceedings of the Twenty-first International Conference on Machine Learning (ICML) (2004).

Li, C.-H., Berner, J., Hong, J.-S., Fong, C.-T. & Kuo, Y.-H. The Taiwan WRF ensemble prediction system: Scientific description, model-error representation and performance results. Asia-Pac. J. Atmos. Sci. 56 , 1–15 (2020).

Chen, D.-S. et al. Impacts of assimilating dual-Doppler radar-retrieval winds on the TWRF predictions of Typhoon Nesat (2017). Terr. Atmos. Ocean. Sci. 32 , 649–667 (2021).

Mardani, M. et al. Residual Diffusion Modeling for Km-scale Atmospheric Downscaling. arXiv , 2309.15214v3, https://doi.org/10.48550/arXiv.2309.15214 (2024).

Han, T. et al. FengWu-GHR: Learning the Kilometer-scale Medium-range Global Weather Forecasting. arXiv:2402.00059v1, https://doi.org/10.48550/arXiv.2402.00059 (2024).

Oskarsson, J., Landelius, T. & Lindsten, F. Graph-based neural weather prediction for limited area modeling. arXiv , 2309.17370v2, https://doi.org/10.48550/arXiv.2309.17370 (2023).

Wang, X., et al Global tropical cyclone intensity forecasting with multi-model multi-scale casual autoregressive model. arXiv , 2402.13270v1, https://doi.org/10.48550/arXiv.2402.13270 (2024).

Alexe, M. et al. ECMWF unveils alpha version of new ML model. Accessed 13 October 2023, https://www.ecmwf.int/en/about/media-centre/aifs-blog/2023/ECMWF-unveils-alpha-version-of-new-ML-model (2023).

Lang, S. et al. AIFS-ECMWF’s data-driven forecasting system. arXiv preprint arXiv:2406.01465 (2024).

Download references

Acknowledgements

The research group of this study appreciates deeply the generality and open minds of the model developers for Pangu-Weather, FCN2, GraphCast, FuXi, and FengWu for providing the codes and instructions in the open domain, allowing us the opportunity to understand and use these models. Five MLWP models are available through the following repositories: Pangu-Weather: https://github.com/198808xc/Pangu-Weather . FourCastNet v2: https://github.com/ecmwf-lab/ai-models-fourcastnetv2 . GraphCast: https://github.com/google-deepmind/graphcast . FuXi: https://github.com/tpys/ai-models-fuxi . FengWu: https://github.com/OpenEarthLab/ai-models-fengwu . We acknowledge the support from the CWA Numerical Information Division, which provided essential computing and storage infrastructure critical for the study. Furthermore, we are grateful to ECMWF for providing the ERA5 reanalysis and IFS forecast fields, which contributed to the research endeavors. Finally, we appreciate two reviewers for their constructive comments and suggestions, which helped us improve the quality of the study and the manuscript.

Author information

Authors and affiliations.

Central Weather Administration, Taipei, Taiwan

Cheng-Chin Liu, Kathryn Hsu, Der-Song Chen, Pao-Liang Chang, Ling-Feng Hsiao, Chin-Tzu Fong, Jing-Shan Hong, Chia-Ping Cheng, Kuo-Chen Lu & Chia-Rong Chen

University of Colorado Colorado Springs, Colorado Springs, CO, USA

Melinda S. Peng

National Taiwan University, Taipei, Taiwan

Hung-Chi Kuo

You can also search for this author in PubMed   Google Scholar

Contributions

C.C.L., K.H., M.S.P., D.S.C., P.L.C., and L.F.H. designed research. C.C.L. produced forecasts of MLWP models. C.C.L. and K.H. wrote the code for data analysis. M.S.P., K.H., C.C.L., and D.S.C. analyzed the data. M.S.P., K.H., C.C.L., P.L.C., and L.F.H. wrote the draft manuscript. C.P.C manages this project. C.T.F., J.S.H., K.C.L., C.R.C., and H.C.K. participated in the discussion and suggested changes to the manuscript. All authors have read and approved the final manuscript.

Corresponding author

Correspondence to Kathryn Hsu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Liu, CC., Hsu, K., Peng, M.S. et al. Evaluation of five global AI models for predicting weather in Eastern Asia and Western Pacific. npj Clim Atmos Sci 7 , 221 (2024). https://doi.org/10.1038/s41612-024-00769-0

Download citation

Received : 11 April 2024

Accepted : 11 September 2024

Published : 28 September 2024

DOI : https://doi.org/10.1038/s41612-024-00769-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

• Foreign Affairs
• CFR Education
• Newsletters

• Securing Ukraine

Climate Change

Global Climate Agreements: Successes and Failures

Backgrounder by Lindsay Maizland December 5, 2023 Renewing America

• Defense & Security
• Diplomacy & International Institutions
• Energy & Environment
• Human Rights
• Politics & Government
• Social Issues

Myanmar’s Troubled History

Backgrounder by Lindsay Maizland January 31, 2022

• Europe & Eurasia
• Global Commons
• Middle East & North Africa
• Sub-Saharan Africa

How Tobacco Laws Could Help Close the Racial Gap on Cancer

Interactive by Olivia Angelino, Thomas J. Bollyky , Elle Ruggiero and Isabella Turilli February 1, 2023 Global Health Program

• Backgrounders
• Special Projects

Conflicts of Interest in Federal Contracting

Testimony by Rush Doshi September 24, 2024 China Strategy Initiative

• Centers & Programs
• Books & Reports
• Task Force Program
• Fellowships

Oil and Petroleum Products

Academic Webinar: The Geopolitics of Oil

Webinar with Carolyn Kissane and Irina A. Faskianos April 12, 2023

• Students and Educators
• State & Local Officials
• Religion Leaders
• Local Journalists

C. Peter McColough Series on International Economics With Lael Brainard

Virtual Event with Lael Brainard and Roger W. Ferguson Jr. March 2, 2021 C. Peter McColough Series on International Economics

• Lectureship Series
• Webinars & Conference Calls
• Member Login

Ukraine: Conflict at the Crossroads of Europe and Russia

• Russia’s unprovoked invasion of Ukraine in 2022 has set alight the bloodiest conflict in Europe since World War II.
• A former Soviet republic, Ukraine had deep cultural, economic, and political bonds with Russia, but the war could irreparably harm their relations.
• Some experts view the Russia-Ukraine war as a manifestation of renewed geopolitical rivalry between major world powers.

Introduction

Ukraine has long played an important, yet sometimes overlooked, role in the global security order. Today, the country is on the front lines of a renewed great-power rivalry that many analysts say will dominate international relations in the decades ahead.

Russia’s invasion of Ukraine in February 2022 marked a dramatic escalation of the eight-year-old conflict that began with Russia’s annexation of Crimea and signified a historic turning point for European security. A year after the fighting began, many defense and foreign policy analysts cast the war as a major strategic blunder by Russian President Vladimir Putin.  

• NATO (North Atlantic Treaty Organization)
• United States
• Military Operations

Many observers see little prospect for a diplomatic resolution in the months ahead and instead acknowledge the potential for a dangerous escalation, which could include Russia’s use of a nuclear weapon. The war has hastened Ukraine’s push to join Western political blocs, including the European Union (EU) and the North Atlantic Treaty Organization (NATO).

Why is Ukraine a geopolitical flash point?

Ukraine was a cornerstone of the Soviet Union, the archrival of the United States during the Cold War. Behind only Russia, it was the second-most-populous and -powerful of the fifteen Soviet republics, home to much of the union’s agricultural production, defense industries, and military, including the Black Sea Fleet and some of the nuclear arsenal. Ukraine was so vital to the union that its decision to sever ties in 1991 proved to be a coup de grâce for the ailing superpower. In its three decades of independence, Ukraine has sought to forge its own path as a sovereign state while looking to align more closely with Western institutions, including the EU and NATO. However, Kyiv struggled to balance its foreign relations and to bridge deep internal divisions . A more nationalist, Ukrainian-speaking population in western parts of the country generally supported greater integration with Europe, while a mostly Russian-speaking community in the east favored closer ties with Russia.

Ukraine became a battleground in 2014 when Russia annexed Crimea and began arming and abetting separatists in the Donbas region in the country’s southeast. Russia’s seizure of Crimea was the first time since World War II that a European state annexed the territory of another. More than fourteen thousand people died in the fighting in the Donbas between 2014 and 2021, the bloodiest conflict in Europe since the Balkan Wars of the 1990s. The hostilities marked a clear shift in the global security environment from a unipolar period of U.S. dominance to one defined by renewed competition between great powers [PDF].

In February 2022, Russia embarked on a full-scale invasion of Ukraine with the aim of toppling the Western-aligned government of Volodymyr Zelenskyy.

What are Russia’s broad interests in Ukraine?

Russia has deep cultural, economic, and political bonds with Ukraine, and in many ways Ukraine is central to Russia’s identity and vision for itself in the world.

Family ties . Russia and Ukraine have strong familial bonds that go back centuries. Kyiv, Ukraine’s capital, is sometimes referred to as “the mother of Russian cities,” on par in terms of cultural influence with Moscow and St. Petersburg. It was in Kyiv in the eighth and ninth centuries that Christianity was brought from Byzantium to the Slavic peoples. And it was Christianity that served as the anchor for Kievan Rus, the early Slavic state from which modern Russians, Ukrainians, and Belarussians draw their lineage.

Russian diaspora . Approximately eight million ethnic Russians were living in Ukraine as of 2001, according to a census taken that year, mostly in the south and east. Moscow claimed a duty to protect these people as a pretext for its actions in Crimea and the Donbas in 2014.

Superpower image . After the Soviet collapse, many Russian politicians viewed the divorce with Ukraine as a mistake of history and a threat to Russia’s standing as a great power. Losing a permanent hold on Ukraine, and letting it fall into the Western orbit, would be seen by many as a major blow to Russia’s international prestige. In 2022, Putin cast the escalating war with Ukraine as a part of a broader struggle against Western powers he says are intent on destroying Russia.

Crimea . Soviet leader Nikita Khrushchev transferred Crimea from Russia to Ukraine in 1954 to strengthen the “brotherly ties between the Ukrainian and Russian peoples.” However, since the fall of the union, many Russian nationalists in both Russia and Crimea longed for a return of the peninsula. The city of Sevastopol is home port for Russia’s Black Sea Fleet, the dominant maritime force in the region.

Trade . Russia was for a long time Ukraine’s largest trading partner , although this link withered dramatically in recent years. China eventually surpassed Russia in trade with Ukraine. Prior to its invasion of Crimea, Russia had hoped to pull Ukraine into its single market, the Eurasian Economic Union, which today includes Armenia, Belarus, Kazakhstan, and Kyrgyzstan.

Energy . Moscow relied on Ukrainian pipelines to pump its gas to customers in Central and Eastern Europe for decades, and it paid Kyiv billions of dollars per year in transit fees. The flow of Russian gas through Ukraine continued in early 2023 despite the hostilities between the two countries, but volumes were reduced and the pipelines remained in serious jeopardy.

Political sway . Russia was keen to preserve its political influence in Ukraine and throughout the former Soviet Union, particularly after its preferred candidate for Ukrainian president in 2004, Viktor Yanukovych, lost to a reformist competitor as part of the Orange Revolution popular movement. This shock to Russia’s interests in Ukraine came after a similar electoral defeat for the Kremlin in Georgia in 2003, known as the Rose Revolution, and was followed by another—the Tulip Revolution—in Kyrgyzstan in 2005. Yanukovych later became president of Ukraine, in 2010, amid voter discontent with the Orange government.

What triggered Russia’s moves in Crimea and the Donbas in 2014?

It was Ukraine’s ties with the EU that brought tensions to a head with Russia in 2013–14. In late 2013, President Yanukovych, acting under pressure from his supporters in Moscow, scrapped plans to formalize a closer economic relationship with the EU. Russia had at the same time been pressing Ukraine to join the not-yet-formed EAEU. Many Ukrainians perceived Yanukovych’s decision as a betrayal by a deeply corrupt and incompetent government, and it ignited countrywide protests known as Euromaidan.

Putin framed the ensuing tumult of Euromaidan, which forced Yanukovych from power, as a Western-backed “fascist coup” that endangered the ethnic Russian majority in Crimea. (Western leaders dismissed this as baseless propaganda reminiscent of the Soviet era.) In response, Putin ordered a covert invasion of Crimea that he later justified as a rescue operation. “There is a limit to everything. And with Ukraine, our western partners have crossed the line,” Putin said in a March 2014 address formalizing the annexation.

Putin employed a similar narrative to justify his support for separatists in southeastern Ukraine, another region home to large numbers of ethnic Russians and Russian speakers. He famously referred to the area as Novorossiya (New Russia), a term dating back to eighteenth-century imperial Russia. Armed Russian provocateurs, including some agents of Russian security services, are believed to have played a central role in stirring the anti-Euromaidan secessionist movements in the region into a rebellion. However, unlike Crimea, Russia continued to officially deny its involvement in the Donbas conflict until it launched its wider invasion of Ukraine in 2022.

Why did Russia launch a full-scale invasion of Ukraine in 2022?

Some Western analysts see Russia’s 2022 invasion as the culmination of the Kremlin’s growing resentment toward NATO’s post–Cold War expansion into the former Soviet sphere of influence. Russian leaders, including Putin, have alleged that the United States and NATO repeatedly violated pledges they made in the early 1990s to not expand the alliance into the former Soviet bloc. They view NATO’s enlargement during this tumultuous period for Russia as a humiliating imposition about which they could do little but watch.

In the weeks leading up to NATO’s 2008 summit, President Vladimir Putin warned U.S. diplomats that steps to bring Ukraine into the alliance “would be a hostile act toward Russia.” Months later, Russia went to war with Georgia, seemingly showcasing Putin’s willingness to use force to secure his country’s interests. (Some independent observers faulted Georgia for initiating the so-called August War but blamed Russia for escalating hostilities.)

Despite remaining a nonmember, Ukraine grew its ties with NATO in the years leading up to the 2022 invasion. Ukraine held annual military exercises with the alliance and, in 2020, became one of just six enhanced opportunity partners, a special status for the bloc’s closest nonmember allies. Moreover, Kyiv affirmed its goal to eventually gain full NATO membership.

In the weeks leading up to its invasion, Russia made several major security demands of the United States and NATO, including that they cease expanding the alliance, seek Russian consent for certain NATO deployments, and remove U.S. nuclear weapons from Europe. Alliance leaders responded that they were open to new diplomacy but were unwilling to discuss shutting NATO’s doors to new members.

“While in the United States we talk about a Ukraine crisis , from the Russian standpoint this is a crisis in European security architecture,” CFR’s Thomas Graham told Arms Control Today in February 2022. “And the fundamental issue they want to negotiate is the revision of European security architecture as it now stands to something that is more favorable to Russian interests.”

Other experts have said that perhaps the most important motivating factor for Putin was his fear that Ukraine would continue to develop into a modern, Western-style democracy that would inevitably undermine his autocratic regime in Russia and dash his hopes of rebuilding a Russia-led sphere of influence in Eastern Europe. “[Putin] wants to destabilize Ukraine , frighten Ukraine,” writes historian Anne Applebaum in the Atlantic . “He wants Ukrainian democracy to fail. He wants the Ukrainian economy to collapse. He wants foreign investors to flee. He wants his neighbors—in Belarus, Kazakhstan, even Poland and Hungary—to doubt whether democracy will ever be viable, in the longer term, in their countries too.”

What are Russia’s objectives in Ukraine?

Putin’s Russia has been described as a revanchist power, keen to regain its former power and prestige. “It was always Putin’s goal to restore Russia to the status of a great power in northern Eurasia,” writes Gerard Toal, an international affairs professor at Virginia Tech, in his book Near Abroad . “The end goal was not to re-create the Soviet Union but to make Russia great again.”

By seizing Crimea in 2014, Russia solidified its control of a strategic foothold on the Black Sea. With a larger and more sophisticated military presence there, Russia can project power deeper into the Mediterranean, Middle East, and North Africa, where it has traditionally had limited influence. Some analysts argue that Western powers failed to impose meaningful costs on Russia in response to its annexation of Crimea, which they say only increased Putin’s willingness to use military force in pursuit of his foreign policy objectives. Until its invasion in 2022, Russia’s strategic gains in the Donbas were more fragile. Supporting the separatists had, at least temporarily, increased its bargaining power vis-à-vis Ukraine.  

In July 2021, Putin authored what many Western foreign policy experts viewed as an ominous article explaining his controversial views of the shared history between Russia and Ukraine. Among other remarks, Putin described Russians and Ukrainians as “one people” who effectively occupy “the same historical and spiritual space.”

Throughout that year, Russia amassed tens of thousands of troops along the border with Ukraine and later into allied Belarus under the auspices of military exercises. In February 2022, Putin ordered a full-scale invasion, crossing a force of some two hundred thousand troops into Ukrainian territory from the south (Crimea), east (Russia), and north (Belarus), in an attempt to seize major cities, including the capital Kyiv, and depose the government. Putin said the broad goals were to “de-Nazify” and “de-militarize” Ukraine.

However, in the early weeks of the invasion, Ukrainian forces marshaled a stalwart resistance that succeeded in bogging down the Russian military in many areas, including in Kyiv. Many defense analysts say that Russian forces have suffered from low morale, poor logistics, and an ill-conceived military strategy that assumed Ukraine would fall quickly and easily.

In August 2022, Ukraine launched a major counteroffensive against Russian forces, recapturing thousands of square miles of territory in the Kharkiv and Kherson regions. The campaigns marked a stunning setback for Russia. Amid the Russian retreat, Putin ordered the mobilization of some three hundred thousand more troops, illegally annexed four more Ukrainian regions, and threatened to use nuclear weapons to defend Russia’s “territorial integrity.”

Fighting in the subsequent months focused along various fronts in the Donbas, and Russia adopted a new tactic of targeting civilian infrastructure in several distant Ukrainian cities, including Kyiv, with missile and drone strikes. At the first-year mark of the war, Western officials estimated that more than one hundred thousand Ukrainians had been killed or wounded , while Russian losses were likely even higher, possibly double that figure. Meanwhile, some eight million refugees had fled Ukraine, and millions more were internally displaced. Ahead of the spring thaw, Ukraine’s Western allies pledged to send more-sophisticated military aid, including tanks. Most security analysts see little chance for diplomacy in the months ahead, as both sides have strong motives to continue the fight.

What have been U.S. priorities in Ukraine?

Immediately following the Soviet collapse, Washington’s priority was pushing Ukraine—along with Belarus and Kazakhstan—to forfeit its nuclear arsenal so that only Russia would retain the former union’s weapons. At the same time, the United States rushed to bolster the shaky democracy in Russia. Some prominent observers at the time felt that the United States was premature in this courtship with Russia, and that it should have worked more on fostering geopolitical pluralism in the rest of the former Soviet Union.

Former U.S. National Security Advisor Zbigniew Brzezinski, in Foreign Affairs in early 1994, described a healthy and stable Ukraine as a critical counterweight to Russia and the lynchpin of what he advocated should be the new U.S. grand strategy after the Cold War. “It cannot be stressed strongly enough that without Ukraine, Russia ceases to be an empire, but with Ukraine suborned and then subordinated, Russia automatically becomes an empire,” he wrote. In the months after Brzezinski’s article was published, the United States, the United Kingdom, and Russia pledged via the Budapest Referendum to respect Ukraine’s independence and sovereignty in return for it becoming a nonnuclear state.

Twenty years later, as Russian forces seized Crimea, restoring and strengthening Ukraine’s sovereignty reemerged as a top U.S. and EU foreign policy priority. Following the 2022 invasion, U.S. and NATO allies dramatically increased defense, economic, and humanitarian assistance to Ukraine, as well as ramped up their sanctions on Russia. However, Western leaders have been careful to avoid actions they believe will draw their countries into the war or otherwise escalate it, which could, in the extreme, pose a nuclear threat.  

Ukraine’s Struggle for Independence in Russia’s Shadow

What are U.S. and EU policy in Ukraine?

The United States remains committed to the restoration of Ukraine’s territorial integrity and sovereignty. It does not recognize Russia’s claims to Crimea or the other regions unlawfully annexed by Russia. Prior to the 2022 invasion, the United States supported a settlement of the Donbas conflict via the Minsk agreements [PDF].

Western powers and their partners have taken many steps to increase aid to Ukraine and punish Russia for its 2022 offensive. As of February 2023, the United States has provided Ukraine more than $50 billion in assistance, which includes advanced military aid, such as rocket and missile systems, helicopters, drones, and tanks. Several NATO allies are providing similar aid.

Meanwhile, the international sanctions on Russia have vastly expanded, covering much of its financial, energy, defense, and tech sectors and targeting the assets of wealthy oligarchs and other individuals. The U.S. and some European governments also banned some Russian banks from the Society for Worldwide Interbank Financial Telecommunication, a financial messaging system known as SWIFT; placed restrictions on Russia’s ability to access its vast foreign reserves; and blacklisted Russia’s central bank. Moreover, many influential Western companies have shuttered or suspended operations in Russia. The Group of Eight, now known as the Group of Seven , suspended Russia from its ranks indefinitely in 2014.  

The invasion also cost Russia its long-awaited Nord Stream 2 pipeline after Germany suspended its regulatory approval in February. Many critics, including U.S. and Ukrainian officials, opposed the natural gas pipeline during its development, claiming it would give Russia greater political leverage over Ukraine and the European gas market. In August, Russia indefinitely suspended operations of Nord Stream 1, which provided the European market with as much as a third of its natural gas.

What do Ukrainians want?

Russia’s aggression in recent years has galvanized public support for Ukraine’s Westward leanings. In the wake of Euromaidan, the country elected as president the billionaire businessman Petro Poroshenko, a staunch proponent of EU and NATO integration. In 2019, Zelensky defeated Poroshenko in a sign of the public’s deep dissatisfaction with the political establishment and its halting battle against corruption and an oligarchic economy.

Before the 2022 offensive, polls indicated that Ukrainians held mixed views on NATO and EU membership . More than half of those surveyed (not including residents of Crimea and the contested regions in the east) supported EU membership, while 40 to 50 percent were in favor of joining NATO.

Just days after the invasion, President Zelenskyy requested that the EU put Ukraine on a fast track to membership. The country became an official candidate in June 2022, but experts caution that the membership process could take years. In September of that year, Zelenskyy submitted a formal application for Ukraine to join NATO, pushing for an accelerated admission process for that bloc as well. Many Western analysts say that, similar to Ukraine’s EU bid, NATO membership does not seem likely in the near term.

• What triggered Russia’s moves in 2014?
• Why did Russia launch an invasion in 2022?

More From Our Experts

How Will the EU Elections Results Change Europe?

In Brief by Liana Fix June 10, 2024 Europe Program

Iran Attack Means an Even Tougher Balancing Act for the U.S. in the Middle East

In Brief by Steven A. Cook April 14, 2024 Middle East Program

Iran Attacks on Israel Spur Escalation Concerns

In Brief by Ray Takeyh April 14, 2024 Middle East Program

Top Stories on CFR

The LDP Leadership Race: Ishiba Shigeru Wins

Blog Post by Sheila A. Smith September 27, 2024 Asia Unbound

U.S. Military Support for Taiwan in Five Charts

Article by Jonathan Masters and Will Merrow September 25, 2024

The Climate Challenge, With Alice Hill and Varun Sivaram (Election 2024, Episode 2)

Podcast with James M. Lindsay , Alice C. Hill and Varun Sivaram September 24, 2024 The President’s Inbox

Resource Catalog

We estimate fine particulate matter (PM2.5) concentrations daily using MODIS satellite observations of aerosol optical depth (AOD) for a major biomass burning event around Moscow during summer 2010. Evaluation of MODIS AOD with the Moscow AERONET site supports a MODIS-AOD error estimate of ±(0.05 + 0.2 x AOD) for this event. However, since the smoke was often thick (AOD > 4.0) and spatially variable, the standard MODIS algorithm incorrectly identifies some aerosol as cloud. We test relaxed cloud screening criteria that increase MODIS coverage by 21% and find excellent agreement with coincident operational retrievals (r2 = 0.994, slope = 1.01) with no evidence of false aerosol detection. We relate the resultant MODIS AOD to PM2.5 using aerosol vertical profiles from the GEOS-Chem chemical transport model. Our estimates are in good agreement with PM2.5 values estimated from in-situ PM10 (r2 = 0.85, slope = 1.06), and we find that the relationship between AOD and PM2.5 is insensitive to uncertainties in biomass burning emissions. The satellite-derived and in-situ values both indicate that peak daily mean concentrations of approximately 600 µg m-3 occurred on August 7, 2010 in the Moscow region of the Russian Federation. We estimate that exposure to air pollution from the Moscow wildfires may have caused hundreds of excess deaths. © 2011 Elsevier Ltd. All rights reserved.

Cataloging Information

• aerosol optical depth
• air quality
• fire management
• Moscow wildfires
• particulates
• remote sensing
• statistical analysis