Is it effective to use AF OUT output as a method for extending the communication range?

Yes. It is effective to put a filter (LPF/BPF) in the AF OUT output in order to extend the communication range. In order to achieve DATAOUT output, a slicer circuit is necessary after the filter. Similarly, lowering the data rate can be expected to extend the distance further.