As they say in the classics: "Use the Source, Luke." . . .
I altered line 863 above (I am using JUCE 5.2.0 so my line number is different) to make the Settings box bigger (600, 854) and this produced a taller dialog box which suits the second change.
The whole section named "Audio and MIDI" is called "viewport" and is produced by JUCE. The AudioDeviceSelectorComponent constructor has no argument for height. In the constructor itself (JUCE-5.2.0/modules/juce_audio_utils/gui/juce_AudioDeviceSelectorComponent.cpp/h), a private data member itemHeight is initialised to 24. This can't be altered by the constructor, but there is a member function setItemHeight(int) to change it. However, this only sets the height of some of the items in the whole block and the spaces between them, not the height of the audio channel list boxes or number of audio channels which can be viewed in them.
The height of the two list boxes for Active output/input channels is set by lines 266 and 273 of JUCE-5.2.0/modules/juce_audio_utils/gui/juce_AudioDeviceSelectorComponent.cpp by calling of a member function of the two list objects whose only parameter is a local variable of the constructor: maxListBoxHeight, which is set to 100. As you wrote, I need to hack the JUCE code to change this value to 200 or more. By changing it to 250 I got list boxes which show 10 pairs of channels, and this suits the 854 high Settings window.