Tory Miller, Madison’s star chef, works hard to balance food, family

February 25, 2018

At Tory Miller’s award-winning restaurant on Capitol Square, L’Etoile, the chef’s tasting menu includes beef tartare with Oscietra caviar and pearl onions and La Belle farms foie gras with banana, miso, cocoa and walnuts.

At home — it’s more likely to be meatballs, tomato sauce and ice cream as Madison’s most celebrated chef races home after 12- to 15-hour days at his restaurants to try to make dinner with his wife, Kristine, and 3-year-old son Miles, or at least arrive before Miles’ 8 p.m. bedtime.

Miller, known for his four high-profile Madison restaurants, L’Etoile, Graze, Estrellon and Sujeo, is drawing national attention and new fans. Already the winner of a James Beard Award (pretty much a culinary Oscar), Miller, 42, took down “unbeatable” Iron Chef Bobby Flay on Food Network’s “Iron Chef Showdown” last month.

With all of his success, Miller, who’s been cooking since age 6, isn’t living in a mansion on Lake Mendota or driving a Mercedes-Benz.

Instead, he and Kristine rent a three-bedroom apartment on Madison’s West Side, near Verona, around the corner from Miles’ day care, and Miller drives a 2014 Chevy Cruze.

“We definitely live a way, way more modest life than anybody would think,” Miller said. “We feel so rich in the things that matter, which are food, and good friends and family. We say that all the time. We’re definitely not rich financially.”

Miller’s long work days are hardly glamorous. And then there’s the drive he makes to Rockford every other week to pick up his older son Remy, 7, who goes to school in Chicago’s Lincoln Park and lives primarily with Miller‘s ex-wife, Lili Calfee. Miller sees Remy every other weekend and also for vacations.

“It’s not super glam,” Miller said about the chef and restaurant-owning life.

To make it work, there’s a lot of juggling, Miller said, “but it’s what you do, and you love to do, and if you have a supporting wife and family then you can make it work.”

Coming to Madison

and L’Etoile

Miller was 29 when he bought L’Etoile, the star of Madison’s restaurant scene, in 2005, with his sister, Traci Miller. Then L’Etoile’s chef de cuisine, or head chef, Miller took over the restaurant from chef-proprietor Odessa Piper, who had owned it for 28 years.

The siblings, who were raised in Racine, formed their Deja Food group right away. In late 2007, they partnered with Dianne Christensen, CEO of the economic consulting company Christensen Associates; and in 2010 moved L’Etoile from its second-floor location at 25 N. Pinckney St. on the Capitol Square, to the all-glass U.S. Bank Plaza building, also on the Square, and at the same time opened the similarly local-food focused gastropub Graze next door.

The restaurant group added two more partners, Tracey Solverson in late 2010, and Krys Wachowiak in 2012, the same year Traci Miller stepped out to focus on her career as a pharmacist. The group established the Asian-focused Sujeo at Livingston Street and East Washington Avenue in 2014, and the Spanish tapas restaurant Estrellón on Johnson Street in 2015.

“He’s incredibly creative,” Christensen said. “It’s absolutely wonderful to see him constantly evolving. The restaurants don’t stand still. The menus change and he comes up with new ideas.”

After graduating second in his class from the French Culinary Institute in New York in 2000, Miller stayed in Manhattan and worked at Bill Telepan’s Judson Grill (the space later taken over by Flay), Eleven Madison Park and Jean Georges.

Impressed by that résumé, Piper hired Miller as a line cook at L’Etoile even though she didn’t have any openings.

Piper quickly observed the kind of quiet, assured humility Miller displayed on “Iron Chef Showdown.”

“He was someone who had so much confidence that he could be very humble and very straightforward,” said Piper, 65, who now works as a restaurant consultant based in Boston.

In part, Miller stood out because he went over and helped the dishwashers blast through the accumulated pots and pans from the morning prep.

“I have a soft spot in my heart for dish washing, because I had to do a lot of it,” Piper said.

Miller worked out so well, he became Piper’s chef de cuisine. She calls it one of the best hiring decisions she ever made.

Growing up in the kitchen

Born in South Korea, Miller was adopted at about 18 months old and doesn’t know his birth parents.

“I didn’t grow up a real Korean,” Miller joked with a cooking class of eight students sitting around the kitchen bar in Estrellón one Monday night last month. Miller was demonstrating how to make Korean bibimbap, something he also made as one of his four dishes on the “Iron Chef Showdown.”

He grew up working at his grandparents’ diner in Racine and remembers heading to the restaurant every day after kindergarten.

“I have vivid memories of being on the step ladder and making food when I was like 6 years old,” Miller said. He began working there legally at 13.

The restaurant, Park Inn, is still there, although his family doesn’t own it anymore. Still, Miller likes to visit, and was there over Christmas with Kristine, Remy and Miles.

Miller’s parents, Bob and Joani Miller, and his brother, Trever, still live in Racine. Traci splits her time between homes in Racine and Madison.

Miller went to college at UW-Stevens Point for two years and dropped out. “I just wasn’t feeling school and I knew that I wanted to be a cook somehow. It had always been a big part of my life,” he said.

He had worked for two years in Stevens Point restaurants when he saw a commercial on the Food Network with Flay and Drew Nieporent (Tribeca Grill, Nobu). They were promoting New York’s French Culinary Institute.

Miller marvels at the irony of that now after his competition with Flay. “I still remember seeing that commercial and I was like, ‘Ah, I’ll look up the French Culinary Institute.’ It was everything I wanted as far as going to New York.”

The yearlong program was more than $25,000, “but it was the opportunity I was looking for,” Miller said.

A showdown 20 years

in the making

Flash forward 20 years to “Iron Chef Showdown,” where Miller took on Flay Jan. 10 while exuding an easy-going humor.

“I wasn’t necessarily calm,” Miller said. “I was definitely excited and I think that comes out in gibberish. I didn’t know what I was saying.”

Cooking with his team, he always manages to have a good time and that comes through, Miller said. “Our profession is also our passion and our joy and our love and it’s what we want to do.”

Itaru Nagano, 40, L’Etoile’s chef de cuisine, said the competition was really stressful at times. Nagano and Desiree Nudd, Miller’s former chef de cuisine at Estrellón, were Miller’s sous chefs in Los Angeles when he taped “Iron Chef Showdown” last summer. Nudd is now a sous chef at Somerset in Chicago.

Because of Flay’s history with judge Giada De Laurentiis, Nagano said he figured they didn’t have much chance of winning. “The odds were kind of stacked up against us,” he said. “We really didn’t know what to expect.”

Nagano calls Miller, a 2012 recipient of the James Beard Award “Best Chef: Midwest,” a great boss. “That’s the No. 1 thing that people should know. He’s one of the best chefs to work for because if you try hard and you put in the effort, he has your back 100 percent.”

Typical day

Most of the time, Miller, despite his growing reputation, isn’t in the limelight.

His typical day starts at Ford’s Gym on Winnebago Street after he drops Miles at day care. He’s been working with personal trainer David McKercher for almost three years, doing a lot of weightlifting.

“Tory’s a super hard worker,” McKercher said. “He’s definitely not in there for any other reason than to just put in some work, get a little bit better, be a little bit healthier. I know he loves it because it’s some time away from that other world — that big food world that he lives in.”

Then Miller checks in at the Deja Food office in the same building that houses Graze and L’Etoile.

After that, he makes his kitchen rounds. He talks to Graze’s chefs first and figures out whether he needs to work lunch or make any menu changes. Then he meets with Nagano at L’Etoile.

He’ll visit Sujeo, and Estrellón, in no particular order, and will typically end up at one of those restaurants for the night. Miller tries to balance his time between all four restaurants, but that can be difficult, he said.

The time Miller spends at home varies, and “is always going to be a struggle,” said Kristine Miller, 32, herself a pastry chef, noting that the time element was hard to adapt to early in their marriage.

Having had her own business, Dough Baby, for 16 months on State Street before it closed in October, makes her understand the demands on Miller.

“Now I understand 1,000 percent everything he has to do,” she said.

Putting down roots

Family is important to Miller and his tattooed arms tell the story. His left arm represents family, his right arm, the restaurants.

When Remy was born, Miller got Remy Joong, his son’s first and middle names, done in Korean characters with his son’s birth date. Because Kristine has Scottish heritage, when Miles was born Miller got the Scottish flag with what Kristine calls a Korean-Scottish lion-dragon and Miles’ initials.

His right arm features various representations of each restaurant, including stars for L’Etoile. He’s in the process of getting a crown for Estrellón, but hasn’t had time to finish it.

These days, fun for Miller is hanging out with Kristine and his boys, making tomato sauce and meatballs and eating ice cream.

Kristine, who is pregnant with another boy, due in April, said they make the most of the time they do have and when they have Remy, Miller tries to work less.

Miller said when he moved to Madison in 2003, he didn’t plan on staying. He thought it would just be a stop along his way.

“I definitely wasn’t planning on living here forever, like I am now.”