Here's a few experiments I ran with shark & fish motion in general to try and make them more believable. It was greatly inspired by the ABZU GDC talk but my results were definitely not up to my likings.
I tried a couple approaches, namely a custom spline interpolation system that would let me drive the shark by generating a new trajectory point at a fixed timestep and then using the spline to deform the shark's mesh using a custom spline interpolation algorithm in a shader.
It had many many issues though, apparent squash & stretch (inherited from the way spline interpolation works), roll issues, bugs I couldn't solve with vertices randomly glitching.
Best solution I had for now is baking curls & twists poses in textures to lerp those poses based on the shark's turn rate but it's not great. Ideally I'd need true support of blendspaces for static meshes like what they did on ABZU.
First a swim animation was authored in Blender and baked into a vertex animation (chase & bite weren't used)
Vertex animation in engine.
It looks okay when shark is static but not so much when the shark turns. It looks very rigid and fake.
In an attempt to fix that, kinda similar to the approach used on ABZU, I baked curl & twist poses in additional textures to bend the shark based on its turn-rate and make the motion look more natural.
Not great, not terrible.
I then experimented with a shader approach using a custom spline interpolation algorithm. It turned unfruitful due to many issues I couldn't solve.
I pass the location of 7 points in world space. 2 location and 2 tangents for the current spline segment and 1 location and 2 tangents for the previous or next spline segment depending on which part of the shark isn't on the current spline segment.
It was hard to figure out... :D I had all kind of bugs and even though I ended scrapping this approach, it was a very interesting challenge to tackle.