After the development of Deepface and DeepID methods in 2014,deep learning methods for image recognition has dramatically improved the state-of-the-art performance on Deep Convolutional Neural Networks（DCNNs）and reshaped the research landscape of image processing and data analysis.In spite of rapid improvement in deep learning algorithms,it still has various challenges like adjustment of appropriate loss function and optimization strategy to handle large scale problems in many computer vision applications including Face Recognition（FR）and Handwritten Digit Recognition（HDR）.This thesis focus on these challenges and their better solution.For both computer vision tasks,there are some advanced approaches based on the Convo-lutional Neural Network（CNN）that are able to learn image features via the softmax loss,but softmax can only learns those separable features that are considered not discriminative enough.A number of modifications set to boost up the discriminative power of the softmax and normally used for encouraging the separability of image features,but these modifications are not suitable for the intra-class variation.On this concern multiplicative angular margin,additive angular margin and additive cosine margin has been introduced to restrict the bound-ary closer to the weight vector of each class but the random selection of marginal values is again a big issue in multiplicative angular margin and additive angular margin.As a solution on this issue,we present a novel approach to handle these problems via presenting an additive parameter relative to multiplicative angular margin for DCNNs and reformulate the softmax loss through combined angular margin and additive margin.Moreover,an automatically fine-tuning is offered to adjust the additive parameter’ as a seedling element growing in the result of marginal seed.Experimental results on additive parameter demonstrate that our approach is better than numerous current state-of-the-art approaches using the similar network architecture and benchmarks.On the other hand,there is no doubt Stochastic Gradient Descent（SGD）avoid spurious local minima and touch those that generalize well,but it decelerates the convergence of regular gradient descent.For linear convergence in strongly convex functions,numerous variance reduction algorithms have been intended,but only few of them are suitable to train DCNNs.A simple modification offered by recently deigned Laplacian Smoothing Gradient Descent（2018）dramatically reduces the optimality gap in SGD and applicable to train DCNNs.Motivated by Laplacian Smoothing Stochastic Gradient Descent（LS-SGD）and inspired from the additive parameter,we adopt this simple modification of gradient descent and stochastic gradient descent to design a novel strategy assembled with a modified form of softmax and LS-SGD.Our approach expresses a flexible learning job with adjustable additive margin and is flexible to amend with SGD and LS-SGD.