Also see: http://forum.unity3d.com/threads/242835-Shader-Forge-Extensions
I made that like 2 years ago and still works :)
I see nested nodes as a way of taking a (huge) set of nodes, call them boiler-plate nodes if you will, and change them to a single node instead. Looking at my clip library I mostly have clips for very repetitive node structures that a bunch of shaders all have in common, and thats what nested nodes should provide IMO. A single node that acts as a bunch of them, to keep the tree small and easily interpretable.
You will still get nested loops of functionA calls functionB which calls functionA again, this is where the stackoverflow exception is for :P You don't have to prevent it at all, just notify the user that stack is getting rather full and that you quit the code generation (I'm pretty sure even GPU's stop processing shaders if they keep calling recursively).
I'd say anything more complex than the standard nodes like add/subtract/oneminus, etc. should be fed to the nested node by means of an input, not as an actual node inside them. It's been a while since I've used UDK/Unreal, but I believe that's how they restrict it. It would make sense, too. Nested nodes don't live on their own, they're not shaders, they're a part of one.
Any statement somewhere as to why SF2 is somewhat canceled?
As for your question: Single in- and output would be pretty limiting, as for a lot of things you would need to feed some extra pre-calculated information into the equation of the nested node. And things calculated inside a nested node, sometimes need their way out as well.
Without properties would be okay, a lot is solved already if you can have multiple inputs as you can just feed the property-data into the nested node that way.
What kind of implication do you forsee when you say you probably wont support nested nodes in nested nodes (this sentence also learns nested nodes isn't a good name either :P We're nesting nested nodes)?
What if you generate include-files out of nested nodes and basically just call a function from the main shader into the include-file? I'd foresee that Unity would recompile it all for you UNLESS you decide to modify the in- and outputs. In your defense, I think even UE4 draws the line there.
If I were to make this: the most flexible way would be to take your Code-node, and instead of code, allow one to assign a sub-shader/nested node/include-file to it and save that node as-is. Then its the nested nodes responsibility to make the inputs/outputs match that of the one defined, and even after it changes, you won't mess up the wiring of all nodes, unless one physically updates/regenerates the node.
Oops! My bad, my client hasn't given me the latest version yet! Just happened to ran across this post in search of where the Fallback-setting was, and saw this happening and figured I'd throw in my stacktrace for you :)
Just managed to trigger it. I clicked "Pick", but didn't pick any shader. Now it will keep throwing the exception each time I click "Pick" again until I restart.
Here is the stacktrace:
UnityEngine.Material..ctor (UnityEngine.Shader shader) (at C:/buildslave/unity/build/artifacts/generated/common/runtime/ShaderBindings.gen.cs:168)
ShaderForge.SFPSC_Meta.DisplayShaderContext (Rect r)
ShaderForge.SFPSC_Meta.ShaderPicker (Rect r, System.String s)
ShaderForge.SFPSC_Meta.DrawInner (UnityEngine.Rect& r)
ShaderForge.SFPS_Category.Draw (Single yOffset)
ShaderForge.SF_PassSettings.OnLocalGUI (Int32 yOffset, Int32 in_maxWidth)
ShaderForge.SF_Editor.DrawPreviewPanel (Rect r)
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:222)
Customer support service by UserEcho