Audio input device issue when device is removed

[coreymaher]

Describe the bug
If the selected audio input device is removed a new device must be selected before the audio input works again. Adding the audio input device again does not resolve the issue.

This is a bigger problem in firefox because if you have 2 input devices and remove 1 you can't get the remaining input device to work without refreshing the page. When you have at least 3 input devices, you'll have 2 left after removing one so you're able to select an input device.

I think this occurs for output devices as well, but I only tested inputs.

To Reproduce
Steps to reproduce the behavior:

1. Start the app with at least 2 audio inputs available on your device (ex: laptop with an internal microphone and external microphone)
2. Open the audio and video settings modal
3. Select the removable audio input device (if not already selected)
4. Remove the audio device
5. Notice the dropdown either becomes empty, or changes the selected value, but no input is captured

Expected behavior
Another input device should be selected automatically and capture input

Screenshots
Chrome Demo - In this video I show what happens when unplugging my mic when it is selected

AudioDeviceSwitchIssue.mp4

Environment (please complete the following information):

• OS: OSX 10.15.7
• Browser: Chrome 89.0.4389.90, Firefox 86.0
• App Version: 0.2.3 (a6fc6c9)
• SDK Version: 2.12.0
• Node.js version: 12.20.1

Additional context
I tried adding logic to AudioInputList to detect when the audio input device list changed and calling localAudioTrack?.restart with either a new device id if the existing one is missing or the same device id if the device id is the same but the group id changed. That did work, but I doubt this is the correct place for this logic since it would only change/restart the input device when the audio and video settings modal is open.

Let me know if there's anything else I can provide.

[timmydoza]

Thanks for the detailed bug report @coreymaher!!

I can confirm that this issue is present in the app. I have created a bug ticket and added it to our backlog.

I'll ask around the team here to see what the best approach might be, but I think we will need to listen to the devicechange event on the navigator.mediaDevices property, as it is fired whenever a device is removed from the system. When this event is fired, we can check to see if the audio track that was published to the room was supplied by the removed device (by checking to see if the device is still returned by navigator.mediaDevices.enumerateDevices()). If it was removed, we can call audioTrack.restart() without a deviceId to capture the system's default audio device and publish it to the room. I think this logic could go somewhere in the VideoProvider component.

[coreymaher]

Thanks for the info @timmydoza! Is there a public backlog or roadmap for this repo (and/or twilio-video in general)? I'd be curious to see which improvements and bug fixes are coming soon and which ones I should attempt to work-around on my own.
Thanks for the great project and all your team's hard work!

[timmydoza]

Thanks @coreymaher!

Twilio video doesn't have a public roadmap, but the bugs found in our github issues will have fixes coming at some point. Generally, if a bug isn't going to be fixed, or a feature request isn't going to be implemented, we will close the issue.

[jszem]

Hi @timmydoza thanks for your responses here. I'd like to +1/bump on this one for prioritization if possible. With WFH, headphones are ever more common. Totally understand if you guys are swamped but would be helpful to understand if a fix could be expected on a scale of days, weeks, or months, so that I can prioritize on my side as well. Thanks again!

[timmydoza]

Thanks for the additional info @jszem!

I was able to consistently reproduce this issue with wired headphones (with mic), so I understand that this issue has the potential to have a significant impact on users. We will bump the priority of the bugfix. It's added to our next sprint, so I expect that the fix should be merged within the next week or two.

[olipyskoty]

Hi @coreymaher and @jszem! This fix has been merged into master 🙂 . Please let us know if you run into any more issues. Have a great weekend!

[coreymaher]

Thanks @olipyskoty! I can't wait for the next release, it looks like a bunch of good features and fixes are going to be part of it!

[coreymaher]

@olipyskoty I had a chance to test out this fix and it works great in Chrome! Unfortunately, it doesn't appear to work in Firefox.
When I unplug my external mic a few errors are printed to the console and it doesn't end up capturing output from the other device.
Here are the errors I'm seeing:

2021-04-21T13:59:25.750Z error [LocalAudioTrack #1: {b8aef587-7138-0e4f-958a-13af62165c48}] Failed to re-acquire the MediaStreamTrack: 
Object { error: MediaStreamError, constraints: {…} }
index.js:1
    e index.js:1
    overrideMethod react_devtools_backend.js:2557
    log log.js:179
    error log.js:269
    _restart localmediatrack.js:208
    (Async: promise callback)
    _restart localmediatrack.js:207
    restart localmediatrack.js:289
    restart localaudiotrack.js:141
    handleDeviceChange useRestartAudioTrackOnDeviceChange.ts:19
    (Async: EventListener.handleEvent)
    useRestartAudioTrackOnDeviceChange useRestartAudioTrackOnDeviceChange.ts:23
    React 6
    unstable_runWithPriority scheduler.development.js:697
    React 4
    unstable_runWithPriority scheduler.development.js:697
    React 5
    getAudioAndVideoTracks useLocalTracks.ts:93
    getAudioAndVideoTracks useLocalTracks.ts:93
    PreJoinScreens PreJoinScreens.tsx:37
    React 6
    unstable_runWithPriority scheduler.development.js:697
    React 4
    unstable_runWithPriority scheduler.development.js:697
    React 6

and

Uncaught (in promise) 
MediaStreamError { name: "OverconstrainedError", message: "Constraints could be not satisfied.", constraint: "deviceId", stack: "" }

Let me know if you need any more info or would like me to open up a separate bug report for this issue.
Thanks again for all the help!

[olipyskoty]

Hi @coreymaher thanks so much for bringing this to our attention. I can reproduce the issue in Firefox exactly as you described. I've added a new bug ticket to our backlog, and we'll let you know as soon as the fix is merged. Have a good weekend!

[coreymaher]

Thanks for taking a look @olipyskoty! Have a good weekend as well!

[olipyskoty]

Hi @coreymaher, thanks again for bringing the Firefox bug to our attention. The fix has been merged into the master branch 🎉. Please reach out if you have any questions or run into any more issues!

[coreymaher]

Thanks so much @olipyskoty!