Rawaudio dev

[dingodoppelt]

Add a new "raw" audio quality setting

This PR adds uncompressed audio ("raw") to the quality settings so there is no Opus compression along the way
Discussion in #3654

This feature improves latency as well. I gained 2ms by using uncompressed audio while having a better audio quality.

CHANGELOG: Add uncompressed audio transmission - dedicated to the memory of Hans Petter Selasky (1982 - 2023)

Does this change need documentation? What needs to be documented and how?
Corresponding PR in jamulussoftware/jamuluswebsite #1133

[EDIT: pljones] Note the above was closed unmerged.

Checklist

• I've verified that this Pull Request follows the general code principles
• I tested my code and it does what I want
• My code follows the style guide
• I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
• I've filled all the content above

[ann0see]

I'd prefer not to check for the Jamulus version number but rather based on capabilities - we don't have 4.0.0 out yet and it might break during the dev process.

[dingodoppelt]

I'd prefer not to check for the Jamulus version number but rather based on capabilities - we don't have 4.0.0 out yet and it might break during the dev process.

I wanted to reuse information already available as much as possible so I just added the code where there were version checks already implemented. (For sequence number and pan feature)
Capabilities would be nice but also would require more changes to client, channel, server and protocol which I don't really have an idea on how to make that backwards compatible. We should rather replace all version checks with some capabilities struct that client and server can agree upon so everything lands in one place. I just don't feel like the right person to take on that challenge and rather pursue my hacky approach, as long as it works for everybody.
The version check with 4.0.0 could be replaced by a point release 3.11.1 and would work right away.

[ann0see]

Tested it and yes, the noise would be unacceptable. What is our fallback if max is selected but the server doesn't support it?

[dingodoppelt]

Tested it and yes, the noise would be unacceptable. What is our fallback if max is selected but the server doesn't support it?

I just noticed that if you connect to a server with Max selected you get the noise unless you switch audio quality again while connected. The server code is fine and doesn't need changes, I misplaced the check for my introduced bRawAudioSupported in the client code. I'll have a closer look
Edit: Funny, the noise doesn't happen on legacy servers, only on rawaudio :D

[dingodoppelt]

Adding a slot to the client to reinit when it receives the server version seems to have fixed the noise issue.

[dingodoppelt]

We still get crashes on windows, especially when using more coplex setups including audio routing software. Linux, Mac and android builds work fine so far. Sounds great but still needs more testing and fixes

[dingodoppelt]

The last commits fixed the crash on windows and make the client fall back to opus reliably. This is now ready to be tested thoroughly.

[dingodoppelt]

A buffersize of 256 on Max quality setting gives garbled audio and the packet sizes seem wrong and contain blocks of zeroes. Only that particular setting is affected. Opus still works

[softins]

A buffersize of 256 on Max quality setting gives garbled audio and the packet sizes seem wrong and contain blocks of zeroes. Only that particular setting is affected. Opus still works

I plan to try out this enhancement over the next few days. I've had a look through the diffs so far. Could you specify exactly the steps to produce this error?

[dingodoppelt]

A buffersize of 256 on Max quality setting gives garbled audio and the packet sizes seem wrong and contain blocks of zeroes. Only that particular setting is affected. Opus still works

I plan to try out this enhancement over the next few days. I've had a look through the diffs so far. Could you specify exactly the steps to produce this error?

This is reproducible with a buffersize of 256 samples only. The packets should be well below MTU and show a length of 1026 in wireshark.
This is may be related to the use of the conversion buffer or the iSndCrdFramSizeFactor not being used in packet size calculation.

Edit: The packets become bigger than the MTU allows for on 256 samples buffersize and get fragmented once I corrected the calculation of the packet sizes. Does this mean we need to disable raw audio for buffersizes of 256 or is there some mechanism to receive fragmented packets?

[softins]

I've just tried a build of rawaudio-dev here, between two separate hosts: server on a pi, client on a PC. It doesn't seem to be a MTU or fragmentation issue. The UDP packets are only 1068 bytes in size, and not fragmented.

Using a buffer size of 10.67ms (256) results in each packet containing two frames of audio, each with its own sequence number. In that setting, I was seeing one packet every 10.67ms coming from the Windows client, but still one packet every 5.33ms coming back from the server. They alternated between having zeros in the first frame and zeros in the second frame. So it could possibly be some issue in server.cpp that doesn't exist in client.cpp

Note that the client will encode according to the settings in the Client Settings dialog, but the server will encode according to the information in received in the NETW_TRANSPORT_PROPS message it received from the client.

Talking of which, the codec field in the NETW_TRANSPORT_PROPS message should specify a different value for RAW, rather than still saying OPUS, like this:

jamulus/src/util.h

Lines 484 to 492 in 849e823

	// Audio compression type enum -------------------------------------------------
	enum EAudComprType
	{
	// used for protocol -> enum values must be fixed!
	CT_NONE = 0,
	CT_CELT = 1,
	CT_OPUS = 2,
	CT_OPUS64 = 3 // using OPUS with 64 samples frame size
	};

So when sending props for raw encoding, it should either use CT_NONE or define a new CT_RAW=4.

[dingodoppelt]

I've just tried a build of rawaudio-dev here, between two separate hosts: server on a pi, client on a PC. It doesn't seem to be a MTU or fragmentation issue. The UDP packets are only 1068 bytes in size, and not fragmented.

This build is not taking into account iSndCrdFrameSizeFactor. From what I understood it should be mostly 1 and my code seems to only work when it is. iCeltNumCodedBytes should be multiplied by iSndCrdFrameSizeFactor. On 256 samples buffer size it will create packets that get fragmented. Wireshark shows the fragmentation. Should I push these changes for you to test? I might have gotten something fundamentally wrong here, but I'd say the problem is mainly in the client since the server happily plays back everything you throw at it.

[softins]

Ah, so the issue is that the client is not sending enough data to satisfy the server, and the server is therefore adding in packets of zeros to maintain the data rate.

Fragmentation should not be an issue, at least with IPv4, as fragmentation and re-assembly happens transparently at the IP layer. In fact, I don't think it will occur anyway, as the traffic from the server is not fragmented. We should just get packets from the client at 5.33ms instead of 10.67ms.

In fact, I've been doing some tests with Wireshark of all the various data rates, qualities and mono/stereo, and it seems that the packet interval is normally half the buffer time specified in the Client Settings. Except when "Small buffers" is not checked, and then 2.67 (64) is exactly the same as 5.33 (128).

[softins]

This build is not taking into account iSndCrdFrameSizeFactor. From what I understood it should be mostly 1 and my code seems to only work when it is. iCeltNumCodedBytes should be multiplied by iSndCrdFrameSizeFactor. On 256 samples buffer size it will create packets that get fragmented. Wireshark shows the fragmentation. Should I push these changes for you to test?

Yes please - I'm building directly from your rawaudio-dev branch.

[softins]

I think in client.cpp around line 1486, you need also to do a similar loop as a few lines above:

for ( i = 0, j = 0; i < iSndCrdFrameSizeFactor; i++, j += iNumAudioChannels * iOPUSFrameSizeSamples )

I don't have any more time today to try it...

[softins]

I think it would be better to recognise the audio frame by a sentinel byte. Protocol frames begin with 00 00 and must have a good checksum. Otherwise they are considered to be audio. Opus frames always begin with 00 for mono and 04 for stereo. So maybe for raw audio, the audio data could be prepended with a byte of f0 for mono and f4 for stereo? Then it could be recognised unambiguously. Both client and server need to recognise the format of a received frame correctly without relying on an out-of-band context.

[dingodoppelt]

As far as I understood these sentinel bytes come from the opus codec itself and are not deliberately set by Jamulus as a message ID of sorts. I'd have to overwrite actual audio bytes for that to work with my code. Or am I wrong here?

[dingodoppelt]

I had misunderstood the packet size calculation and it seems fixed with the last commit.

[dingodoppelt]

Note that the client will encode according to the settings in the Client Settings dialog, but the server will encode according to the information in received in the NETW_TRANSPORT_PROPS message it received from the client.

Talking of which, the codec field in the NETW_TRANSPORT_PROPS message should specify a different value for RAW, rather than still saying OPUS, like this:

I think OPUS and OPUS64 only refer to the setting of small network buffers. It isn't related to the actual opus coding.

[softins]

Note that the client will encode according to the settings in the Client Settings dialog, but the server will encode according to the information in received in the NETW_TRANSPORT_PROPS message it received from the client.
Talking of which, the codec field in the NETW_TRANSPORT_PROPS message should specify a different value for RAW, rather than still saying OPUS, like this:

I think OPUS and OPUS64 only refer to the setting of small network buffers. It isn't related to the actual opus coding.

Maybe - I hadn't got around to examining how the value was used in the code. It just felt wrong for the message to state OPUS when it wasn't, and maybe a specific value could also be useful to the server.

[dingodoppelt]

Maybe - I hadn't got around to examining how the value was used in the code. It just felt wrong for the message to state OPUS when it wasn't, and maybe a specific value could also be useful to the server.

The server isn't aware of the opus quality setting. It is only being sent the packet size and feeds that into the opus codec. There is currently no other way for the server than to determine the codec (or none) by the expected packet sizes for rawaudio. OPUS and OPUS64 refer to 128 or 64 samples internal buffering, no relations to audio quality settings.

[softins]

Just tried the latest build. It's looking good in Wireshark and sounding good too. No fragmentation either, as the max packet size is only 1068 for stereo, max quality, 10.67ms(256).

[JoshuaDodds]

Have you guys also tested with small network buffers? This seems to work as well for me and with a huge improvement on latency as well (as expected). Tested on arm64 ubuntu server side and win11 for the client.

[pljones]

Hi @dingodoppelt,

Jamulus 3.12.1 has been released -- could you rebase, please.

Thanks,

-- Peter

[ann0see]

How do you all feel of this PR? Can we soon get it in?

[foobarth]

How do you all feel of this PR? Can we soon get it in?

I can't say anything bad about these dev builds. Raw audio works well for me, apart from the now more audible jitter glitches, which is acceptable. Have been using this build since the early -rawaudio versions on macOS and several Linux distros and never ran into stability or quality issues.

The only thing that haunts me from time to time is the already reported malloc() crash on macOS when exiting the app. So far, i have not been able to reproduce it reliably nor track down the root cause or if it is even related to this PRs code.

[pljones]

I think it needs squashing to three commits -- original commit (as is), single commit for the new protocol message, single commit (as is) for the new command line option.

[dingodoppelt]

I think it needs squashing to three commits -- original commit (as is), single commit for the new protocol message, single commit (as is) for the new command line option.

Did that. @softins I hope the commit message is fine for you, otherwise I'll change it to your liking.

[DeMuirs]

Curious about this. Does this happen with the dmg Mac installer or is it the legacy installer for Mac? Would be good to know.

…

On 19/05/2026 18:05, Ralf Barth wrote: *foobarth* left a comment (jamulussoftware/jamulus#3653) <#3653 (comment)> How do you all feel of this PR? Can we soon get it in? I can't say anything bad about these dev builds. Raw audio works well for me, apart from the now more audible jitter glitches, which is acceptable. Have been using this build since the early -rawaudio versions on macOS and several Linux distros and never ran into stability or quality issues. The only thing that haunts me from time to time is the already reported malloc() crash on macOS when exiting the app. So far, i have not been able to reproduce it reliably nor track down the root cause or if it is even related to this PRs code. — Reply to this email directly, view it on GitHub <#3653 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BMOHOHHSQ43GCSGV3NA363L43SH4NAVCNFSM6AAAAACX5OYMPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIOJQGE2DGOBTHA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[softins]

I think it needs squashing to three commits -- original commit (as is), single commit for the new protocol message, single commit (as is) for the new command line option.

Did that. @softins I hope the commit message is fine for you, otherwise I'll change it to your liking.

Yes, that's fine. Thank you for asking and for all the hard work!

[softins]

Happy to approve. Would appreciate reviews from both @ann0see and @pljones as some of the code is my own.

[foobarth]

Curious about this. Does this happen with the dmg Mac installer or is it the legacy installer for Mac? Would be good to know.

Default DMG installer on macOS 26.4/5, M1 Pro.

[pljones]

I'd like an issue raised for some refactoring of the names to be more meaningful throughout the audio code. I think that's part of what makes it hard to understand: the names distract from what's actually going on.

[ann0see]

It's in now... I would not yet tag a beta - I'd like to get the .app changes in and potentially TCP.

[dingodoppelt]

Thank you so much for all the code review and input @softins @pljones @ann0see @corrados @foobarth @rdica
You all made this past month was a very pleasant one for me and the merge is the cherry on top :)
(We only missed Hans' birthday by five days, btw ;)

[pljones]

OK, very late to the party...

Connection flow: (C=..5; S=..19)

...lots of ping exchanges...
C->S a packet of sound (ideally we'd start the connection with a packet of mono silence in 128 byte low OPUS encoding, just to wake the server...)
S->C CLIENT_ID (0)

C->S NETW_TRANSPORT_PROPS

So the client goes "I'd like Stereo OPUS with small frames" and the server goes "OK, but I do RAW_AUDIO". Then the client goes, "hm, fine, I could have asked for that to start with" and sends

C->S NETW_TRANSPORT_PROPS

with CODEC still OPUS64 but a different Base Netw Size.

That still feels a bit skewed.

What would a server that didn't understand a new Codec ID have done? Presumably not sent back a normal ACK. The new client could have used that to downgrade gracefully.

[softins]

Messages that are not CLM_xxx are always acked as soon as they are received. The ACKN is only to verify receipt, not semantic validity. Even message types that are not understood are acked.

The server informs the client of its ability to do raw audio at the first opportunity. There is no way for the client to know this before that message. Although I'm surprised that in your example the NETW_TRANSPORT_PROPS is sent by the client so early. It didn't use to be sent until the server asked for it, but when the RAWAUDIO_SUPPORTED message was introduced that also caused the NETW_TRANSPORT_PROPS to be sent in response, but I hadn't seen the really early one before.

Also, although OPUS and OPUS64 currently only manage the buffer size, I think it would make more sense to have different values for codec (RAW and RAW64 or PCM and PCM64) to indicate that the requested stream is raw/PCM rather than OPUS.

Any breaking changes in this should be made between 3.12.1dev and 4.0.0. We will need to maintain compatibility between versions, servers and clients more strictly once we release 4.0.0, so it's worth taking the time to get it right now.