We propose a new asymmetric pixel confusion algorithm for images based on the Rivest-Shamir-Adleman (RSA) public-key cryptosystem and Arnold map. First, the RSA asymmetric algorithm is used to generate two groups of Arnold transform parameters to address the problem of symmetrical distribution of Arnold map parameters. Second, the image is divided into blocks, and the first group of parameters is used to perform Arnold confusion on each sub-block. Then, the second group of parameters is used to perform Arnold confusion on the entire image. The image correlation is thereby fully weakened, and the image confusion degree and effect are further enhanced. The experimental results show that the proposed image pixel confusion algorithm has better confusion effect than the classical Arnold map based confusion and the row-column exchange based confusion. Specifically, the values of gray difference are close to one. In addition, the security of the new confusion operation is dependent on RSA, and it can act as one part of a confusion-substitution structure in a cipher.