The Vibrato is horrible to work with.
I believe I know what it's doing, but I don't quite have all the figures yet to back up my suspicion.
The amount of phase shift introduced by the vibrato, has to be proportional to the frequency of the sound. In fact the chip uses an approximation, and it's quite a broad approximation at that.
The FM LFO has a wavelength of 8192 samples, and operates in eight different ways, each for 1024 samples. If I draw up a little list of values, and then go on to explain them, each value is applied for 1024 samples, then it starts again.
+2, +3, +2, 0, -2, -3, -2, 0
In my list, the amplitude of the number represents how many significant bits of F-NUM are added or subtracted to make the phase modulation, the sign of the number indicates whether that's an addition or a subtraction.
So for each of the first 1024 samples, the F-NUM value is modified by adding to it the most significant 2 bits (i.e. the 10-bit F-NUM is shifted 8 bits to the right and then added to F-NUM)
For the next 1024 samples, the F-NUM value is modified by adding to it the most significant 3 bits (i.e. the 10-bit F-NUM is shifted 7 bits to the right and then added to F-NUM)
In later samples, the same values will be subtracted rather than added, which means that the waveform drifts in and out of it's original phase over the 8192 sample wavelength.
Note that the above figures are for the greater depth option, the documentation says this is 14%. For the 7% option the shifts are 1 bit further, (+1, +2, +1, 0, -1, -2, -1, 0)
Note that for F-NUM < 128 there is no discernable vibrato at all. This is perhaps significant because you can produce a note from an F-NUM < 128 which is the same frequency as another note with F-NUM > 128 by changing the BLOCK value. But although they are the same frequency, they will show different levels of vibrato.
The MAME emulator does all of this using a 128 entry lookup table. The lookup depends on the most significant 3 bits of F-NUM, the DVB and the 10th bit of the LFO. The resultant looked up value is added to the increment prior to the BLOCK and MULT shifting. I'm sure that this lookup is efficient, and may be the best approach for efficiency, but it doesn't explain where the figures come from. I think that the figures in that lookup table can be explained by what I've described above.
Because this phase modulation depends on F-NUM, it has to be calculated per-sample. Unlike the AM tremolo, for which two values can be calculated for the entire device just once every 64 samples, the VIB values have to be calculated per operator, per sample.