In this paper, we introduce two convergent Monte Carlo algorithms for optimizing complex stochastic systems. The first algorithm, which is applicable to regenerative processes, operates by estimating finite differences. The second method is of Robbins-Monro type and is applicable to Markov chains. The algorithm is driven by derivative estimates obtained via a likelihood ratio argument.