In order to establish a restoring-force model for modified concrete columns with recycled aggregates concrete (RAC), cyclic loading tests were carried out on five concretes with RAC columns and ordinary concrete frame columns under the combined influence with different admixtures and admixtures ratios (silica fume and hybrid fiber). The expressions for characteristic nodes of the skeleton curve were given by the analysis and numerical regression of the test results. In addition, the hysteretic rules of the restoring-force model and the expression for unloading stiffness were presented. Finally, we summed up the complete calculation method of the hysteretic restoring force, whose results were in good agreement with experiment. The results demonstrated that the proposed model could simulate and reflect the corresponding hysteretic behaviors, and the calculation method can provide the theoretical basis for the engineering application.