When Yahoo! alerted employees recently that it would essentially ban them from working from home, CEO Marissa Mayer came under instant fire from all quarters. What all these trigger-happy critics seemed to miss, however, was that Mayer was right. Working from home is a privilege, not a right.

Related: Working from Home Is Patriotic

I’ve been working from home for over 20 years, so this is the rare topic on which I can claim some expertise. When I first started doing so, working from home was rare, and to this day my parents believe I’m infinitely available all day during the week, unlike those who travel into an office each day. That’s a source of frustration, but more frustrating were our early attempts at obtaining mortgages. My wife was essentially solely responsible for our first home—I believe the bank in Phoenix viewed me mostly as a kept man—and by the time we purchased our second home, I had to supply an unusual amount of paperwork and back taxes.

Today, working from home is much more common and certainly more socially acceptable, although images persist of people working in pajamas and attending virtual meetings in their underwear. Oddly enough, although I became an employee of Penton, the Windows IT Pro parent company, a few years back, the one thing that didn’t change at all was that I continued working from home. I travel, of course, and I occasionally show up at the corporate offices in Fort Collins, Colorado, like some crazy-eyed homeless person. But for the most part, my day-to-day situation hasn’t changed much over the years. I work. From home.

Related: Blurring the Line Between Work and Home

I have learned a lot in doing so, of course. Working from home isn’t for everyone, even for those people who, like me, might benefit from the lack of distractions it provides. So even if you have the personality to handle this work style, and you have the type of job in which doing so makes sense for whatever reason, it’s not for everyone. Some might prefer or even require the more regimental schedule of the workplace, or the regular human interaction.

That’s all understandable, but I’m not super-concerned about the wants of employees, to be honest. The bigger issue, as I see it, is that working from home is only sometimes possible—not everyone’s job can be performed offsite—and is also a privilege. For all the namby-pamby “consumerization of IT” and “bring your own device” to work silliness that pervades IT today, topping it off with a whiny new class of knowledge workers demanding to work at home sort of boggles the mind. We’ve gone from a nation of builders and doers to a services-oriented society that doesn’t even want to get out of bed in the morning. It has to stop somewhere.

And that’s where Yahoo! comes in. See, for all the righteous indignation over Mayer’s decision to call home the troops, in this case she was right to do so. Yahoo! is, after all, a failing company, one that hasn’t released a meaningful new product or service for years. What the heck are all those employees doing, anyway? If I were running the show at Yahoo!, I’d scale back on working from home as much as possible. At the very least, I’d tie it to performance metrics.

“To become the absolute best place to work, communication and collaboration will be important, so we need to be working side-by-side,” a very reasonable memo to Yahoo! employees explains. “That is why it is critical that we are all present in our offices. Some of the best decisions and insights come from hallway and cafeteria discussions, meeting new people, and impromptu team meetings. Speed and quality are often sacrificed when we work from home. We need to be one Yahoo!, and that starts with physically being together.”

Over time, it’s possible and even probable that Yahoo! will loosen the reins a bit. But given the company’s dire prognosis today, requiring employees to come into the office and be accountable for their jobs isn’t just OK, it’s the responsible thing to do.

And remember, there’s something far worse than losing your ability to work from home: Losing your job.