For as long as consumer electronics have been tied to the holiday season cycle, engineering heroism is what got those devices on shelves. There is incredible pressure to deliver high quality devices, in abundant quantity, on schedule in order to meet revenue targets. To make it happen, engineers become heroes: last-minute flights, all-nighters, hand carries, and strokes of genius are all baked into these epic deliveries. I was an engineer at Apple
AAPL
for nearly six years, delivering four different products, and the stories of last minute heroics were lauded, told and retold. Heroism is a great way to motivate engineers, but is a terrible risk for companies relying on new gadget sales to make their annual revenue targets.
When I was a hardware leader, I never considered how much risk these heroics added to our programs.
In normal times, August is when engineering and manufacturing teams put the finishing touches on their assembly instructions, test suites, and manufacturing processes and look to replicate lines, increase their production output from ten per hour to hundreds, and ultimately build units for customers. It’s during those final days in August when the heroics happen: vendors driving all night to hand-deliver a box of parts, the engineer MacGyvering a solution to a functional issue with an elaborate assemblage of glue and tape, or the last-minute invention of a test that can finally distinguish good and bad units so the line can start again.
These teams pride themselves on these heroics, and their ability to meet the schedule. But I don’t need to tell you that these are not normal times.
Covid-19 hit Wuhan right before Chinese New Year, a time of massive human migration throughout the country, and also the time that many companies were planning to be just completing or just starting their first development build (as part of a process often called New Product Introduction, or NPI). A typical consumer electronics development cycle consists of three main builds of prototypes and ends with units for paying customers. These builds are a fury of issue discovery and iteration activities because electronic products are complex and there’s only three builds to get it all right. To make it happen, engineers will fly thousands of miles from their headquarters to be onsite in factories typically in China and Greater Asia.
Except, this year, they couldn’t.
With most hardware development engineers grounded since February, this has been a strange spring. Companies and teams had to reinvent how they do development. We surveyed multiple teams to see how they managed their first spring of “manufacturing from home”, and how their first summer of “remote ramps” are shaping up. What follows are glimpses into how three consumer electronics development teams are getting new gadgets ready for fall releases.
Remote Control Buddy
When engineers travel to support builds in the factory, they often spend their time doing failure analysis: the careful process of trying to diagnose what caused a failure to happen and how to prevent building more failures like it. One engineer told us how his team was doing “eight hour FaceTime calls” with factory-side engineers. Calling from his couch at home in California, the engineer would ask his “factory buddy” to walk to certain locations on the assembly line, use the phone’s camera to zoom in on fixtures or units with failures, and try to provide instructions about what next steps to take.
The engineer said it was slow-going – slow to find problems and slow to fix them. “Doesn’t that get old?” I asked. They scoffed, “Some of our leaders built products at Motorola, they are hard core and love eight hour calls.” This tactic is so common that there’s been a rush on talent in the world’s manufacturing centers: both factories and brands are adding in-region project managers and engineers to their rosters as fast as they can to try to handle the demand for on-the-line support.
Night Shift and One Way Trips
Engineers at one Californian company described a multi-week planned shift to work “China time.” They start their day around 6PM Pacific, just in time for the 9AM China-time daily kickoff call, and continue through until 5AM Pacific, which is when the day-shift engineers in China head home. Engineers we spoke to are doing this for two or more weeks at a time, six days per week, in an attempt to remotely support builds they cannot fly to. The individual engineers feel like they are working double-shifts, particularly those with children or other family obligations during the day. The company landed on this arrangement after first trying to find volunteers on the team who might be willing to sign up for a “one-way trip” to China. Takers would do a two week quarantine upon entry to China, and then move on to support build activities. The engineer we spoke to described the trip as “one-way” because of significant doubt about being able to easily return to the U.S. after the build. There were no takers.
Takin’ it To The Streets (Literally)
Even if you can get to the factory, being allowed inside is another question. One small, US-based electronics company scrambling to meet increased demand for its product in the current environment thought they would be in good shape. They started the spring with an established engineering team in China. But factories are taking Covid-19 concerns very seriously, and many have instituted policies prohibiting even their customers from coming inside. Unable to enter, the local engineering team set themselves up in a hotel room across the street. When there’s an issue on the assembly line, such as a new defect, or something causing a high failure rate, their factory-side teammates will carry half-assembled devices outside of the factory to meet with them in the street to get feedback and advice on what corrective actions they should take.
Heroes are Great, But Needing Them is Bad
Many of the most advanced electronics devices we love are built with processes that are rudimentary, luck-based, and brittle. Engineering and operations leaders, focused on delivering great products on time, rely too much on the brilliance and drive of a few individual engineers to “make it work” in the final days leading up to their scheduled production start.
But this reliance is brittle: what happens when your heroes cannot be heroes?
Product margins and revenue at risk, the loss of peak days of sales for the holiday season can quickly add up to hundreds of thousands, millions, or even tens of millions of dollars. Spring 2020 was the wake up call: this risk is too big for executives to ignore. Heroism is an anti-KPI and technology can help. Better software tools and better data enables teams to work ahead of the challenges, potentially even remotely. Sure, hitting your schedule with a smooth ramp doesn’t feel so glamorous in the re-telling, but that’s exactly the problem.