In the forthcoming 5G systems, new technologies, such as amorphous networks and non-orthogonal filter bank multicarriers (FBMC), provide an effective way to accommodate high-rate transmissions. Meanwhile,the prototype filter affects the adjacent channel interference, and therefore is important for FBMC systems. Besides, once the amorphous network is taken into account, the requirement for interference controlmust be much stricter.Accordingly, this paper focuses on the design of prototype filter with better ability of interference controlling, where we exploit the nonlinear phase FIR filter (NLPFF) instead of traditional linear phase FIR filter (LPFF)to achieve more optimization spaces under a small sacrifice of linear phase. In ourdesigns,both the amplitude and phase responsesare handled independently to approach the stopband performance enhancements, in while the nearly perfect reconstruction (NPR) conditionsare relaxed by pre-specified thresholds. Computer simulations confirm the effectiveness of the NLPFF designs, and demonstrate the advantages of the proposed NLPFF in FBMC applications.