This paper explores some of the reasons that recognizing the embedded Reber grammar (Cleeremans et al 1989) is such a difficult problem for Simple Recurrent Networks and proposes an extension to SRNs in an attempt to simplify the learning task. The analysis shows that although the SRN architecture is, in principle, capable of retaining distant past information by utilizing large recurrent weights, the small initial weights prevent the network from learning long term dependencies in the data. The particular extension proposed is to periodically lock the activation of some of the hidden units for a certain number of time steps. Experimental results show that although networks with periodic units do not learn the long term dependencies of the grammar in the majority of trials, they do achieve a significant performance improvement over SRNs. In some cases, networks with periodic units learn the embedded Reber grammar almost perfectly. Further experiments explore the generality of the architecture by evaluating its ability to recognize other regular languages.