It all begins with the leaf texture creation. There's many ways to do this, I tend to model a single leaf in 3D then group a bunch of them together like so.
Then I build my leaf cards out of these clumps.
Here I also added some extra leaves at the edges to not only use this texture for the leaf cards but also as a tilable texture
I bake a mask texture in Blender using a simple shader: it combines the opacity mask in red channel, depth in green and random value per leaf in blue channel
A normal map is also easily generated using a simple shader in Blender
The tree mesh is built using three simple inflated shapes to create a "layered shell". You can save some triangles on the inner shells.
In the first UV map I unwrap those shells the traditional way, nothing fancy here. There will be a visible seam but it's not super noticeable. There's trick to hide such seams but here I kept the material simple, so we'll have to live with it.
In the second UV map I used a simple projection from side view to bake the height (useful to create local gradients in the shader!). I also scaled each "shell layer" uvs to 0 in X and shifted them in the UV map so I can build a "shell gradient"
Always mess with the normals when it comes to stylized rendered foliage :D Here I used a simple sphere to project the normals and have a smoother looking shape
Place some leaf cards here and there to build a more interesting silhouette and make the tree fluffier. Use the same custom normal trick as shown above.
Here again, for the cards, I used a side view UV projection for the second UV map to bake the height. Then shifted randomly each leaf card in X to bake a random value (which may be useful to drive some colors & randomization in the shader)
The second UV map in the end looks like this. It's split in two so I can easily mask cards from the shell meshes using simple maths...
... like so! I built a bunch of masks and gradients out of that second UV map. Here I can isole cards from the inner mesh and drive some logic with it, like here lerp a debug color.
That "CardsMask" is used to sample my mask texture in two different ways for the opacity mask. As-is for the cards and in a tiling manner for the shell meshes (thus having made the texture not only an atlas for our cards but a tilable texture as well)
Now I don't want the most inner shell to still have some masked pixels so I'm building yet another mask using the second UV map to isole that most inner shell...
... and use it to force an opaque material on that last shell. I don't want to be able to see-through the foliage, however dense it is because I may be able to see the tree trunk poking through the bottom which may not look great.
I also derive a shell gradient from that second UV map. Very useful to drive some colors and make the inner part of the tree darker to give it depth and add a fake occlusion effect.
I also have that top/down local gradient! Very useful to drive colors.
And voilĂ , use those gradients & masks to drive albedo & SSS colors. Add a fake rim light and you're done.