So my last blog post got a surprising amount of attention, not least from a/the (no clue how many there are of those) Product Manager of Visual Studio, which is pretty neat.
So, here’s a quick followup, in order to fully exploit my 15 minutes of fame!
First, I highly recommend any newcomers read Doug Turnure’s comment on my previous post. It is an interesting read, and seems surprisingly frank about the problems with Connect as it is today. And where I would normally have expected some canned PR speech about how all the perceived problems aren’t actually problems, or can’t reasonably be fixed, it actually states that the problems are real, and that they will be fixed. Thanks, Doug!
Of course, words are cheap, and it remains to be seen whether any of this will actually result in tangible improvements to Connect. But knowing that someone within Microsoft is actually working on it is a lot better than nothing.
But being the cynical and demanding bastard that I am, I worry that this is only brushing the surface. Because the problem is not just that bugs get forgotten on Connect. Let’s try to separate the issues a bit:
Bugs are forgotten
As we saw, bugs are left to languish until they are mentioned on Twitter, implying that the way to get a bug fixed is not to bother with Connect, but instead to make a lot of noise about it on social media. Doug answered this quite clearly, saying that bugs currently “fall through the cracks”, which can, should and will be prevented in the future. That’s great, but it doesn’t solve the second part of the problem:
Even when bugs don’t fall off the radar, the way they’re handled is still broken
Bugs seem to be treated completely randomly once they’re entered into Connect. Even if they don’t “fall through the cracks”, they are still treated to a line of over-zealous moderators or whatever they are, who seem to be competing on who can close the most issues the fastest. Bugs are closed haphazardly, sometimes as a duplicate of a bug reported on an internal system that customers can’t see, sometimes as “won’t fix”, and sometimes as “won’t fix”, but with a comment saying that it may get fixed for a future release (which means it’s not quite “won’t fix” after all).
Sometimes the same scenario will result in a “deferred” status (which makes a lot more sense than “won’t fix”), and sometimes the bug will be closed as “by design”. Sometimes they’re Resolved as won’t fix, and other times they’re closed as won’t fix. What’s the difference? Are statuses assigned completely at random, or does it just look that way?
Why are issues closed when the team doesn’t have time to fix them in the upcoming release? They’re still bugs, and until a decision is made to never fix the bug, it is hard to see why they should not be considered as open, or unresolved, at least from the users’ point of view (what status it is given internally is of course up to Microsoft. But when customers see a bug closed as “won’t fix” merely because “we don’t have time to fix this in time for the next release”, they’re going to see red, and for good reason.)
Is any or all of this a consequence of the messy internal systems that Doug described, or is it that new bugs are processed and cataloged badly (whether due to the responsible staff receiving insufficient training, the policies they’re supposed to follow being unclear, or the tools they use being, well, broken)? I don’t know, but something is clearly wrong. And it is not just that bugs fall through the cracks and are forgotten.
So the real problems go deeper…
What needs to be done
Once the “superficial” problems with Connect are fixed, the next step is to make sure bugs reports are actually handled correctly: that bugs aren’t closed with “won’t fix”, unless that is actually what the responsible team intends; that duplicate bug reports are closed either with the same reason, or marked as duplicates, and that in general, reporting a bug gives the user the impression that “the bug report isn’t participating in some random lottery, where a random person will eventually close it with a random reason”.
But even then, we’re not quite home free. Because even when a bug is “fixed”, there’s a significant problem:
It hardly ever is!
When a bug is marked as “fixed“on Connect, it usually means “we have fixed this bug internally, in our trunk, and at some unspecified point in the future, we will release some unspecified product built from this source tree”.
In other words, the bug has not been fixed in the product where it was reported. The bug may have been reported in Visual Studio 2010, but it was “fixed” in Visual Studio 2012 (or whatever).
That’s not what “fixed” means, is it? A bug is fixed when it is no longer present in the product where it was reported. In other words, if a bug was reported against VS2010, then I’d consider it “fixed” when the following conditions are true:
- the bug has been fixed in the VS2010 source tree, and
- the fix is available to customers who own VS2010, or at the very least, has been scheduled for availability to users of VS2010.
Of course, some bugs are so insignificant that it’s not worth pushing an update out to users of the current product, and sometimes a fix would be so complex that it can’t be reasonably implemented in the product. Of course there are cases where it makes sense to implement the fix in the trunk of your source tree, and not in the product version where it was reported. But at least give it a different status than “fixed” then.
When a bug is “fixed” in Connect today, the message to the person who reported the bug basically boils down to this:
We have identified the problem, and if you buy a different product, priced at several thousand dollars1, to be released in a year or two, you won’t experience it any more”.
Gee, really? So where does Connect enter into it? I’m pretty sure I can do that today without ever submitting the bug report to Connect. I could just buy a different OS or IDE. I hear a lot of people are pretty happy with XCode, for example.
Of course, exacerbating this problem, Visual Studio upgrades tend to be nontrivial for large code bases. The format of solution and project files is generally incompatible between versions (even when the only actual change to the format is an incremented version number), so not only does the user have to wait for a completely separate product to be released, and then buy it at the usual price, they also have to go through a potentially painful upgrade process… Just to see the fix to a bug they reported two years ago.
Even if all the other problems are fixed, this one still ensures that reporting bugs on Connect will feel like a waste of time. If we’re practically guaranteed that the product we reported the bug against won’t ever be fixed, why bother reporting the bug at all?
So it’s not just Connect
So this plays into another of my pet peeves about Visual Studio: I have no clue whose decision this is, and if it is actually an active decision made by someone who thinks it is a good idea, or if it is simply a fossilized holdover from the mid-80’s that no one have thought to change, but the policy of never ever improving on a released product has got to go. If I report a bug on VS2010 and you claim to have fixed it, I want the fix. If you don’t actually fix VS2010, then you haven’t fixed the bug that I reported. And if you’re not going to fix the bugs I report, where’s my incentive for reporting them?
Of course, once we accept that bugs should actually be fixed, when practically possible, it is worth asking why the same shouldn’t apply to new features and other improvements.
If you have a better compiler than the one I’m using, it really seems to be a win-win situation if you let me use it. I get to use a better compiler, I get more value out of the product I bought, and I am more inclined to buy VS vNext as well, if I know that it too will actually improve over time. Forcing me to use an, on average, 18 months old compiler doesn’t seem like good sales tactics.
Of course, not all VS users are affected equally by this. When I worked with .NET, this bothered me less, because the VS compiler set the pace: the language got updated when the compiler said so, and there were no alternative implementations to speak of. Who cares if your compiler is 2 years old? The language evolves in lockstep with the compiler anyway. Sure, I’d still have loved to see improvements to the IDE itself, but I knew there would be little point in updating the compiler out of band.
But now that I work with C++ every day, it drives me nuts. The VC++ team is (like any other C++ compiler team) perpetually playing catch-up to an international standard, and to a half-dozen competing compilers. When VC10 was released, it was on par with the competition for about 2 months. Sure, it lacked plenty of features that other compilers supported, but it also supported some that weren’t yet implemented in other compilers. Then, a couple of months later, a new GCC version was released which supported the few features that had previously been unique to MSVC. And after that, we’re witnessing a 2–3 year period of utter stagnation, where the VC++ compiler stands completely still, while all around it, the competition is making huge strides towards compliance with the new C++ standard, and making new features available by the dozen.
How is that a good idea? How does this make Visual Studio look good? Why would I pay thousands of dollars for a product that is obsolete 95% of its lifetime?
And the product doesn’t even need to be obsolete at all. The VC++ team is busy implementing countless improvements to the IDE and the compiler. But they’re being “hoarded”, waiting for the next major release, rather than being shipped out to your paying customers. It is as if someone at the top of Microsoft’s DevDiv considers software to be like fine wine; as if new features should be given a few years in a cold cellar to “mature” before being sold to customers.
Does anyone really think that this makes VS a more attractive product?
Connect is current broken, sure, but the problem goes deeper than a few “forgotten” bugs. Once that is fixed, there seems to be a lot of work to do in making Connect appear worthwhile to users (by giving better, more consistent, feedback on received bugs, and probably also through general site improvements, fixing the UX disaster that it is today), and even once those are solved, end-users are unlikely to notice much actual improvement, unless some changes are made to how the Visual Studio team in general deal with bugs, features and product updates.
At the end of the day, until we, the end-users actually get to benefit from the improvements made as a result of of our bug reports, the system will remain broken.
You can make Connect as responsive, user-friendly and all-round wonderful to use as you like — but if we, the end-users, don’t get to see fixes to the bugs we report, the system is still broken.
Of course, my ranting about this is unlikely to change much. I’m hardly in a position to tell Microsoft how to run their business. But on the other hand, there can be no harm in pointing out what I see as problems. So here we go. :)
-
yes, most people and companies are able to get pretty major discounts on that, but not all, and it’s the principle of it that bothers me. ↩
