Denial-of-Service (DoS) attacks are virulent to both computer and networked systems. Modeling and evaluating DoS attacks are very important issues to networked systems; they provide both mathematical foundations and theoretic guidelines to security system design. As defense against DoS has been built more and more into security protocols, this paper studies how to evaluate the risk of DoS in security protocols. First, we build a formal framework to model protocol operations and attacker capabilities. Then we propose an economic model for the risk evaluation. By characterizing the intruder capability with a probability model, our risk evaluation model specifies the Value-at-Risk (VaR) for the security protocols. The Value-at-Risk represents how much computing resources are expected to lose with a given level of confidence. The proposed model can help users to have a better understanding of the protocols they are using, and in the meantime help designers to examine their designs and get clues of improvement. Finally we apply the proposed model to analyze a key agreement protocol used in sensor networks and identify a DoS flaw there, and we also validate the applicability and effectiveness of our risk evaluation model by applying it to analyze and compare two public key authentication protocols.