[Ardour-Dev] how to precisely align the audio output of a Jack MIDI client in Ardour?

What version of Ardour is involved?

Post by Markus Grabner
Hi!
I recently made some effort to measure the latencies of my audio equipment to
get perfectly aligned capturing and playback with Ardour (version 5.11), and
so far it worked well. However, I failed at a task which I thought should be
one of the easiest. I have a Jack client which takes MIDI input and produces
audio output (actually LinuxSampler, but the same issue can be demonstrated
with the attached minimal Jack client and Ardour session). When recording the
audio output of the Jack MIDI client into an Ardour audio track, I noticed
that the signal is delayed by one Jack period, regardless of the latency set
by the client for its ports. When I change the alignment mode of the audio
track to which I'm recording from "Automatic (Capture Time)" to "Align With
Existing Material", the signal is recorded too early by (m-1)*n+o frames,
where m is the number of periods, n the frame size, and o the output latency
of the Jack server. Latency of the Jack MIDI client does have an effect now,
but only shifts the signal even earlier in time.
*) In "Align With Capture Time" mode, the audio signal created by the Jack
MIDI client is too late, and latency settings of the Jack MIDI client are
ignored.
*) In "Align With Existing Material" mode, the audio signal created by the
Jack MIDI client is too early, and can only be shifted to an even earlier time
by setting the latency of the Jack MIDI client.
So what is the recommended way to get frame-accurate recordings of the audio
output of a Jack MIDI client receiving input from an Ardour MIDI track (aside
from manually nudging the recorded regions)?
Thanks & kind regards,
Markus
_______________________________________________
ardour-dev mailing list
http://lists.ardour.org/listinfo.cgi/ardour-dev-ardour.org

Markus Grabner

2017-11-10 00:03:00 UTC

Post by Paul Davis
What version of Ardour is involved?

I first noticed this in Ardour 5.11.0 (built from unmodified git sources, tag
"5.11"), but I didn't investigate it in detail in earlier versions, so I can't
tell when this issue appeared. I'll have a look at Ardour 5.12 tomorrow.

Kind regards,
Markus

Post by Paul Davis

Markus Grabner

2017-11-10 05:18:56 UTC

Post by Paul Davis
What version of Ardour is involved?

I first noticed this in Ardour 5.11.0 (built from unmodified git sources,
tag "5.11"), but I didn't investigate it in detail in earlier versions, so
I can't tell when this issue appeared. I'll have a look at Ardour 5.12
tomorrow.

Ardour 5.12 crashes on startup with a segmentation fault, so I can't test this
right now.

Kind regards,
Markus

Robin Gareus

2017-11-10 15:37:59 UTC

Post by Markus Grabner
Ardour 5.12 crashes on startup with a segmentation fault, so I can't test this
right now.

Would you mind to investigate this and get backtrace
(https://ardour.org/debugging_ardour)?!
Also try with a new empty session to rule out any plugin issues.

Thanks in advance.
robin

PS. For the issue at hand this won't make a difference. There as been no
change in Ardour 5.x for alignment and latency compensation.

Markus Grabner

2017-11-10 18:19:14 UTC

Post by Markus Grabner
Ardour 5.12 crashes on startup with a segmentation fault, so I can't test
this right now.

Would you mind to investigate this and get backtrace
(https://ardour.org/debugging_ardour)?!
Also try with a new empty session to rule out any plugin issues.

After I first experienced the crash, I downgraded to Ardour-5.11, now I just
installed 5.12 again to produce the backtrace, and interestingly its gets a
bit further this time. The application starts and opens the dialog which lets
me choose one of the recently open sessions. However, immediately after
selecting one, it crashes again. The backtrace is attached, I hope it contains
all the information you need.

Kind regards,
Markus

Paul Davis

2017-11-10 19:03:57 UTC

where are you getting ardour from ardour.org ?

Post by Markus Grabner
Ardour 5.12 crashes on startup with a segmentation fault, so I can't

test

Post by Markus Grabner
this right now.

Would you mind to investigate this and get backtrace
(https://ardour.org/debugging_ardour)?!
Also try with a new empty session to rule out any plugin issues.

Markus Grabner

2017-11-10 20:07:48 UTC

Post by Paul Davis
where are you getting ardour from ardour.org ?

I cloned the git repository git://git.ardour.org/ardour/ardour.git, checked
out tag 5.11 (and 5.12) and compiled it from the sources (more precisely, I
took openSUSE's RPM spec file for Ardour-5.0.0 and dropped in the 5.11 sources
instead - worked almost out of the box).

Kind regards,
Markus

Robin Gareus

2017-11-10 04:56:00 UTC

Hi Markus,

It looks like you're sending and receiving data in the same jack client.

your client -> Ardour -> your client.

This cannot happen in the same process cycle. Jack will break the cycle
and either run Ardour first or your client first. So a delay of 1 cycle
is expected.

ciao,
robin

Markus Grabner

2017-11-10 05:26:25 UTC

Post by Robin Gareus
Hi Markus,
It looks like you're sending and receiving data in the same jack client.
your client -> Ardour -> your client.

No, it's actually

Ardour (MIDI track) -> my client -> Ardour (audio track),

and finally it should be

Hydrogen -> LinuxSampler -> Ardour,

where I first noticed the issue, but stripped it down to the simpler case I
reported yesterday. Both cases (with 2 or 3 different clients involved) behave
exactly the same w.r.t. the one period delay.

Post by Robin Gareus
This cannot happen in the same process cycle. Jack will break the cycle
and either run Ardour first or your client first. So a delay of 1 cycle
is expected.

No problem with this, but how can this be compensated?

BTW, my version of jack is 1.9.10.

Kind regards,
Markus

Robin Gareus

2017-11-10 15:36:07 UTC

Post by Robin Gareus
Hi Markus,
It looks like you're sending and receiving data in the same jack client.
your client -> Ardour -> your client.

No, it's actually
Ardour (MIDI track) -> my client -> Ardour (audio track),

Same thing.
I'm sorry I should not have used arrows. They're easily confused with
port connections. Jack runs clients in a given order. Perhaps this is
better represented vertically and without arrows:

cycle 1:
1.1: midipulse
1.2: ardour

cycle 2:
2.1 midipulse
2.2: ardour

cycle 3:
3.1: midipulse
3.2: ardour

etc

Let assume there is a midi-note at time 0 and Ardour starts rolling at
time 0 in cycle 1, with a jack period of N samples.

cycle 1: midipulse runs first. it's midi-input is empty and hence
there's no output. Then Ardour runs and records N samples of silence and
plays a MIDI note at time t = 0.

cycle 2: midipulse runs. sees the MIDI-event (in this cycle at sample 0)
and produces audio: a spike at 0 in this cycle. Ardour runs and records
the produced audio. The spike ends up an N on the global time-line.

The same would be true if the graph was reversed:

cycle 1:
1.1: ardour
1.2: midipulse

cycle 2:
2.1: ardour
2.2: midipulse

C 1.1 Ardour records N samples of silence, sends a MIDI-event
C 1.2 midipulse processes the midi-event and produces audio.
C 2.1 Ardour records the generated audio. 1 cycle late.

As for Ardour's alignment modes.

"Align With Capture Time" is only to be used for internal Ardour
bounces. Ardour 5 aligns the Clock with the Input. The clock represents
the time at which something is recorded; hence "capture time". If you
record an Ardour track onto another Ardour Track, they run in the same
cycle. External port-latencies are not relevant.

"Align With Existing Material" assumes Ardour plays back, the signal
goes outside Ardour, someone plays along, Ardour records that and aligns
it with the material that was played.

Ardour playing takes at least one cycle plus the worst-case latency of
all port-connection to the outside: "Worst playback latency".
Recording in the next cycle is late by the capture latency of the port
that you record. So Ardour 5 moves the recorded signal back (towards 0)
by "worst playback latency" + "port capture latency".

I guess when you did your test, you still had other tracks or busses,
probably ardour's master-bus connected to physical output and
"worst-playback latency" was not zero.

Your simple midipulse jack client ignores any port-latency. Check with
`jack_lsp -l`. In case there are hardware connections, midi_pulse:out
does have a non-zero playback latency. "How long will it be until the
signal reaches the output"; that's the (m-1)*n+o in your original
post. The simple client ignores this and produces audio assuming it's
zero. But even if that simple client would properly takes the
port-latencies into account, the result would still be one cycle off.

I am not aware of a jack-mechanism to that sets port-latencies to
include the 1 cycle latency of the split graph.

Also jack does not automatically compensate for latency (it just passes
numbers around to allow clients to do so).

in case of
no-input -> Hydrogen (MIDI in H2) -> LinuxSampler -> Ardour -> output

it can work if you use jack-transport. The corresponding graph above can
be executed in a single cycle, and all clients (H2, Ardour) share a
common transport.

but the circular graph
X -> Hydrogen -> LinuxSampler -> Ardour (MIDI from A) -> X
will again be broken in two and a 1 cycle latency will occur.

c'est la vie. It's arguably a bug or design mistake that jack does not
set port-latencies when it splits the graph to include the split [1] or
vice-versa that it doesn't delay signal "in the wire" for internal
connections [2].

[1] set the capture latency of the midipulse audio-port in the first
example to include the split or [2] delay the midi signal by 1 cycle in
the second example above. Either would yield the same result.

ciao,
robin

Markus Grabner

2017-11-10 20:16:48 UTC

Post by Robin Gareus
Hi Markus,
It looks like you're sending and receiving data in the same jack client.
your client -> Ardour -> your client.

No, it's actually
Ardour (MIDI track) -> my client -> Ardour (audio track),

Same thing.
[...]

Thanks for the explanation!

Post by Robin Gareus
in case of
no-input -> Hydrogen (MIDI in H2) -> LinuxSampler -> Ardour -> output
it can work if you use jack-transport. The corresponding graph above can
be executed in a single cycle, and all clients (H2, Ardour) share a
common transport.

I tried with the following connection graph:
no input -> Hydrogen -> midipulse -> Ardour -> no output

i.e., the audio interface was completely disconnected, Hydrogen receives no
input, and Ardour produces no output. This gave the same results, i.e., it
makes no difference whether midipulse receives its MIDI input from Ardour or
Hydrogen.

However, when I switch to "Align With Existing Material" and set the audio
capture latency of the midipulse client to one period, the recorded pulses are
perfectly aligned.

Well, at least technically we found a solution, but it's not a practical one
since it would require to modify LinuxSampler (to report the capture latency)
and to disconnect all audio before recording the output of a MIDI client in an
audio track (and to reconnect after).

Post by Robin Gareus
c'est la vie. It's arguably a bug or design mistake that jack does not
set port-latencies when it splits the graph to include the split [1] or
vice-versa that it doesn't delay signal "in the wire" for internal
connections [2].
[1] set the capture latency of the midipulse audio-port in the first
example to include the split or [2] delay the midi signal by 1 cycle in
the second example above. Either would yield the same result.

It seems that jack doesn't process MIDI and audio in the same cycle, even if
each client appears only once (no cyclic graph). How could this be fixed? Could
Ardour examine the connection graph, and if it finds a MIDI/audio transition,
add a latency of one period? What would you suggest?

Thanks & kind regards,
Markus

Markus Grabner

2017-11-11 22:21:44 UTC

I agree that it's a bug, but I don't see anything in the design of jack which
would prevent fixing this, although I believe only option [2] (internal delays)
could be a generic solution for arbitrarily complex connection graphs.

Actually I found another instance of this issue in my audio setup. I initially
stated that I managed to compensate latencies of my devices, but I only
checked it for each device separately. Now after our conversation, I repeated
the tests with both devices simultaneously, and indeed latency compensation
was incorrect.

Let me describe my setup in some more detail:
*) a Focusrite Saffire PRO 14 for microphone and line input and for monitor and
headphone output
*) a Line6 PODxt Live for guitar recording, guitar monitoring, effects
processing, and reamping
*) Zita-ajbridge for conversion between the Focusrite sampling rate (I choose
48000Hz) and the weird hard-coded sampling rate of the Line6 device
(39062.5Hz)

The reamping feature is particularly cool :-), it allows recording the clean
guitar sound while hearing the processed signal, then mixing and cutting the
clean signal (still hearing the processed signal), and finally, when satisfied
with the mix, processing the entire clean guitar track and recording the
processed signal into a regular audio track.

I determined the latencies with Jack_delay, and recording a series of pulses
in either of the following configurations (connections are from top to bottom)
confirmed frame-accurate latency compensation:

1.) Ardour track (playback)
2.) Focusrite in (Firewire)
3.) Focusrite out (6.35mm jack)
4.) audio cable
5.) Focusrite in (6.35mm jack)
6.) Focusrite out (Firewire)
7.) Ardour track (capture)

and

1.) Ardour track (playback)
2.) zita-j2a (sample rate conversion 48000Hz -> 39062.5Hz)
3.) Line6 in (USB)
4.) Line6 internal effect processing
5.) Line6 out (USB)
6.) zita-a2j (sample rate conversion 39062.5Hz -> 48000Hz)
7.) Ardour track (capture)

However, if both signal paths (through the Focusrite and Line6 devices) are
active simultaneously, only the larger latency (involving the sample rate
conversions) is compensated correctly, while the signal traveling through the
Focusrite arrives too early (due to the smaller latency along its path).

So even if simultaneous reamping and playback/recording of other signals is
not really a relevant use case, it is at least inconvenient to work with since
I would have to disconnect and reconnect Jack ports whenever I switch between
recording and reamping (such that Jack only sees a single signal path, and
latencies can be compensated correctly).

What can we do about this? I think it can't be solved outside of jack if
parallel paths in the connection graph have different latencies. But if jack
reports the maximum latency of all paths a client is connected to (as it
already does), it is sufficient to internally delay the signal on the "faster"
path if two paths with different latency arrive at the same node.

I understand that Jack is optimized for low latency operation, but iMO out-of-
the-box support for complex recording setups with frame-accurate latency
compensation would be a very useful addition to its feature set. It's
certainly not a quick fix, but I don't see a reason why it can't be done, and
clients like Ardour wouldn't even have to be changed. What do you think about
it?

Kind regards,
Markus

P.S.: I'm a C++ software developer in my daytime job and willing to help
implementing this (or any other suitable) proposal.

P.P.S.: This discussion transformed from an Ardour question into an (almost)
pure Jack issue. Shall we continue on the Jack mailing list?

Robin Gareus

2017-11-12 00:41:06 UTC

Jack's design goal is to only provide mechanisms (not policy), so (2) is
out. Also either change would break semantics of the existing API.

Jack users have lived with this issue for over a decade: A circular
dependency between clients will split the process graph. Changing jack's
behavior now will be rather tricky. It would require a major version
bump, ideally also intentionally breaking the API because the behavior
will be different compared to jack1, 2.
... or you may want to contribute to pipewire.org instead.

Personally I'm not in favor of adding delay-lines to jack, mostly
because jack can't know what is the correct way to handle dynamic
latency changes, nor can it handle ambiguous latencies.

IMHO, the only case where it would make sense for jackd to delay a
signal is when summing to align physical (terminal) output ports in case
of ambiguous playback port-latency.

[A] output: here, playback to hardware; from a jack-client's POV they're
writable inputs.

Anyway, this would indeed be rather something for jack-devel.

As side-note: the idea to add support for "delay in the wire" in
Ardour's backends (and drop JACK support) has been on the table a while
back.

Post by Markus Grabner
*) a Focusrite Saffire PRO 14 for microphone and line input and for monitor and
headphone output
*) a Line6 PODxt Live for guitar recording, guitar monitoring, effects
processing, and reamping

Is that a USB device? If so the systemic latency may be different every
time you use the device (That's why Ardour's ALSA backend allows to
measure and set systemic latencies without engine restart).

Post by Markus Grabner
*) Zita-ajbridge for conversion between the Focusrite sampling rate (I choose
48000Hz) and the weird hard-coded sampling rate of the Line6 device
(39062.5Hz)

Do you patch the Line6 using an Ardour Port Insert?
That allows to measure and set dedicated latencies for the insert-path.

[...]

Post by Markus Grabner
What can we do about this?

As for Ardour, the solution will be rather simple: Don't use JACK :)

Multiple devices are planned (there's a working prototype for Ardour's
ALSA backend but no GUI to configure it yet. It's currently accessible
via the ALSAEXT env variable) on OSX it's already possible, thanks to
CoreAudio.

Also instead of external apps, prefer plugins. That also saves a context
switch, cannot lead to process-graph-splitting and the plugin's state is
saved with the session.

2c,
robin

Markus Grabner

2017-11-18 22:02:13 UTC

Post by Robin Gareus
IMHO, the only case where it would make sense for jackd to delay a
signal is when summing to align physical (terminal) output ports in case
of ambiguous playback port-latency.

The jack_lsync tool by Xavier Mendez can do just this (thanks for his hint on
the Jack-devel mailing list, see https://github.com/jmendeth/jack-tools/tree/
latency-tools).

After fixing the latencies for the external hardware ports, I went on to
reinvestigate the initial issue of misaligned output of LinuxSampler running
as a Jack-MIDI client. However, the problem has magically disappeared as soon
as jack_lsync is operating on the hardware ports, although the ports related
to LinuxSampler were not even touched in any way. Lucky, but strange, maybe
due to different internal processing order and/or cycle breaking?