Cooperative and defective behavior depends on economic rules (i.e., game theory) and follows psychological laws (e.g., Prospect theory and other social psychology).In this paper, we proposed a two-stage approach to probe into the process of individual behavior decision: non-rational and Prospect theory embedded rational decision.Furthermore, to capture the underlying mechanism of polarization and reversal of collective behavior in the smart environment, modeling of catastrophe in collective behavior is designed by integrating cusp catastrophe model and QSIM (i.e., a qualitative simulation method).Verifications and validations show the modeling results are suitable for explaining the dynamics of collective behavior in the smart environment.