Time-delay neural networks: representation and induction of finite-state machines

1 September 1997

journal article
research article
Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks

Vol. 8 (5) , 1065-1070
https://doi.org/10.1109/72.623208

Abstract

In this work, we characterize and contrast the capabilities of the general class of time-delay neural networks (TDNNs) with input delay neural networks (IDNNs), the subclass of TDNNs with delays limited to the inputs. Each class of networks is capable of representing the same set of languages, those embodied by the definite memory machines (DMMs), a subclass of finite-state machines. We demonstrate the close affinity between TDNNs and DMM languages by learning a very large DMM (2048 states) using only a few training examples. Even though both architectures are capable of representing the same class of languages, they have distinguishable learning biases. Intuition suggests that general TDNNs which include delays in hidden layers should perform well, compared to IDNNs, on problems in which the output can be expressed as a function on narrow input windows which repeat in time. On the other hand, these general TDNNs should perform poorly when the input windows are wide, or there is little repetition. We confirm these hypotheses via a set of simulations and statistical analysis.

Keywords

This publication has 8 references indexed in Scilit:

Grammatical Interference: Learning Syntax from Sentences
Published by Springer Nature ,1996
Learning a class of large finite state machines with a recurrent neural network
Neural Networks, 1995
On the node complexity of neural networks
Neural Networks, 1994
Grammatical Inference and Applications
Published by Springer Nature ,1994
A time-delay neural network architecture for isolated word recognition
Neural Networks, 1990
Phoneme recognition using time-delay neural networks
IEEE Transactions on Acoustics, Speech, and Signal Processing, 1989
Parallel Distributed Processing
Published by MIT Press ,1986
Inductive Inference: Theory and Methods
ACM Computing Surveys, 1983