Mind the gap

It’s interesting to reflect on how long it takes some features to go from the idea stage to being a finished feature you can use in released software. It has been frequently noted that building software is not like building bridges. You often don’t know upfront what the implementation will look like, or what unexpected issues will derail the project, or what nasty bugs will appear that take a long time to work around.

In between the idea stage and the complete stage you often have an intermediate step: a test or preview of the feature where you can see it doing whatever cool thing it does. In my experience, that intermediate step usually sits much closer to the idea stage than the completion stage. I think that this is often confusing for users of the software though; they saw a video of the feature “working”, so why didn’t that feature go into the next release? Sometimes years go by and the feature is still not released. What the user doesn’t often see is the work that goes into making a feature stable and making the code maintainable.

To make maintainable code requires writing code that fits nicely with the existing software’s architecture. If that architecture can’t accommodate the new feature nicely, it’s best to make intermediate changes the build up groundwork needed for the new feature. Incidentally, this is one thing I find very helpful about my rebasing workflow in git. You might notice that the first few commits I make for larger features such as sculpt masking are often code cleanups and other non-sculpt-related changes.

Many of the features that I’ve worked on for Blender have gone through a number of re-implementations. For example, I first coded sculpt masking while working on SharpConstruct. I think I tried at least once or twice to implement masking in Blender 2.4x. I later rewrote it for Blender 2.5 during one of my GSoC projects, but I didn’t feel that code was quite good enough to go into trunk — it just didn’t have that feeling of being “obviously correct”. So I rewrote it again this year, and this time the code felt right. Even that final rewrite took months though. There is a huge benefit to getting the code right, and that’s fewer bugs to deal with. I think only a few bugs have been reported about sculpt masking so far, and all were pretty simple fixes (knock on wood.)

I include that example because after every GSoC, there’s inevitably a certain amount of disappointment from users about “failed” projects. Some express derision for GSoC as a whole, noting that so many projects never seem to make it into trunk. While it’s true that some projects really do completely fail (student disappears, or simply doesn’t have the necessary coding chops), I think it’s good to give them the benefit of the doubt. They might yet figure out how to make the project work. Even if not, they’ve probably learned a lot about Blender’s code and might contribute other features if they see the Blender community as a welcoming and forgiving environment. And indeed, I think that Blender’s users are overall nice about delays and flaws in the development process, so thanks for that :)

This has gotten a bit too long and rambly! So: next up I’ll discuss a bit more the state of SPR and some other features that are in that nebulous gap between idea and completion.

21 thoughts on “Mind the gap”

  1. Awesome blog post, I think this should be required reading for anyone who’s ever wondered (myself included):
    “Why is X feature taking so long to get into trunk?”

  2. Couldn’t have said that better! Knowing the code and making the right changes before extending it is very crucial! Hopefully blender has some nice features code-wise that make this easy to handle for parts of the new features like GUI code.

  3. This is very insightful. I hope this encourages more devlelopers to have regularly updated progress blogs, like yours here, since it makes the “gap” more tolerable for users like myself who have almost no understanding of the software development process.

    Regarding projects in the “gap”, would you mind elaborating on the state of your image overlay project? You have a couple vimeo videos showing this feature in use for ptex and vertex painting modes, and I vaguely recall reading somewhere that this feature is on hold for two reasons:

    (1) It is designed for ptex painting, which itself is on hold.
    (2) Further developing it for vpaint is time wasted because ptex essentially makes vpaint obsolete.

    While these a very valid reasons, why not develop it for texture paint mode, which ptex would not deprecate? Moreover, image overlay in sculpt mode, where the value of each pixel would affect the brush intensity, would surely prove to be a useful addition to Blender. As an example, Mudbox uses it’s own version of image overlay called “stencils” for ptex painting, uv texture painting, and sculpting.

    1. I agree that, lacking Ptex, texture paint is the right place for it. I did briefly look into porting it there, but quickly got confused by the texture paint code. I might look into it again, or perhaps Psy-Fi would be interested? I’d love to see general code cleanups and code documentation for texpaint actually, I always get lost there.

  4. Meaningful post Nicolas … btw. I’ve found sculpt masking very useful for terrain sculpting, thank you for your effort

  5. I completely agree: what seems like the last 10% of the work always takes most of the time. Great post, and I (as with the rest of the Blender community) really appreciate your hard work!!

  6. Hi Nicholas,

    I am a developer (.net techonoliges) myself not a blender developer though, i completely understand what you mean. Every time there is a blender release i feel that you guys are great and awesome because every time there are lot of fixes making it more and more stable from what it was and at the same time include something new or the new cool feature. I am following blender for the past 5 years and i have seen many cool features but not all came real soon but most of them made it to current blender release and i am sure rest of them will also make it steadily and i will wait for them.

    And once again thank you for your contribution to the blender. You blender developers are great.

    And one more thing your post has made me think about learning python and look into blender API. I will start spending some time everyday thanks once again.

  7. thank’s for this precise explain :-).
    In the blenderclan.org (frech site) we found good personne to speak of this too..and now y can explain for this to :-).

    That said, I take this opportunity to thank you for your involvement in the developement of blender.
    thank you, and all the coders who make things happen.

    ++

  8. The problems you describe can easily be fixed and alleviated by a good planning and outlining process. When it comes to programming I tend to be old school: plan out your program (top-down, flowchart), write the pseudocode, and then start coding.

    Today’s approach seems to be come up with an idea and then immediately start coding. And also a lack of style and cohesion among programmers and coders!

  9. Wow. This is an awesome blog. Just found it through blendernation. A treasure trove of valuable development info. Gotta read through it all. Great article! Thanks for this!

  10. I have a simple suggestion, and have made it before…not everyone has an $7000 Origin Genesis. Add an Opengl32.dll to the Blender file, and Intel graphic cards will no longer crash the program when they open the User Preferences. Done deal. I’m sick of hearing about this problem, it’s so easy to fix. Got this fix from BlenderUnderground by the way, by way of Jabhacksoul, so thank him not me :D

      1. Thank you for your reply! Apparently it’s specific to Intel drivers, which is why the user prefs goes ape when selecting it. The bug is already published, and they said they were working on it, but I have yet to see it as even 2.63a’s prefs crash to this day :(
        This is what happens with no opengl32.dll in the file by the way:
        http://imageshack.us/f/715/51072786.jpg/

        1. It might just be that the Intel drivers have an issue with opening two OpenGL windows. As a workaround, you could try opening the user prefs in the main Blender window by switching the space type.

    1. As Mr. Bishop mentions, “OpenGL32.dll” is hardware specific. Placing a version in the Blender folder simply overrides the version that already exists on your system. (Check the ..WindowsSystem32 directory… you’ll probably find a different version there already on your system that was installed as part of your vid card installation) The new one overrides the old, previously existing one due to the way the Win OS searches for calls to DLL’s… “home dir of the calling exe first”… so Blender openGL calls are intercepted by the new DLL in the Blender dir before the OS goes searching thru the default PATH environment variable list of directories… “System32” is one of the dir’s in the “PATH”… and finds the notoriously crappy OpenGL implementation in Intel’s default OpenGL32.dll that already exists there in the system32 dir.

      IOW… YOUR version of this driver will not work with MY hardware, nor anyone else with a different card than yours. By all accounts (I read thru the Blender Underground posts), it works for you out of shear luck. That, or the version you have is actually a “software driver” version (as opposed to the “hardware accelerated” version installed as part of the driver installation), perhaps the MESA library version… basically a software-only open-source implementation of the OpenGL library.

      http://www.mesa3d.org/

      Another issue is that “OpenGL32.dll” is a 32-bit driver. no worky on 64- bit OS.

      1. Thanks for the reply! I see what you’re saying, thank you for explaining it. But, the fact that it works at all is saying something. I’m no programmer/comp guru, but perhaps someone can find a workaround to make everyone happy, and not call for everyone to simply upgrade their computers. The guy at BU came up with this as a workaround for something entirely different in 2.49, and it sure worked for me in my case. Opening the user preferences and have the words and half the buttons disappear is quite frustrating let me tell you :D
        Thanks again!

Leave a Reply

Your email address will not be published. Required fields are marked *