Rearrange FDs which have errored out in listener's FD info arrays to avoid polling.

The process for removing FDs has been adjusted so that active and
inactive FDs remain contiguous at the beginning and end (respectively)
of the l->fds and l->fd_info arrays.