A triple $(W,G,F)$, where $W$ is a topological space, $G$ is a topological group and $F$ is a continuous function $G\times W\to W$ defining a left action of $G$ on $W$: If $w\in W$, $e$ is the identity element of $G$ and $g,h\in G$, then (using multiplicative notation for the operation in $G$) $F(e,w)=w$ and
$$F(gh,w)=F(g,F(h,w))\label{1}\tag{1}$$
(in other words, if one denotes the transformation $w\to F(g,w)$ by $T_g$, then $T_{gh}=T_gT_h$). Instead of a left action one often considers a right action. In this case the arguments of $F$ are usually written in the other order (expressing $F$ as a mapping $W\times G\to W$), and \eqref{1} is replaced by the condition
$$F(w,gh)=F(F(w,g),h).\label{2}\tag{2}$$
Instead of $F(g,w)$ or $F(w,g)$ one often writes simply $gw$ or $wg$. Then \eqref{1} and \eqref{2} are written in the form
$$(gh)w=g(hw),\quad w(gh)=(wg)h.$$
If $G$ is commutative, then there is no essential difference between a left and a right action. The most important cases are $G=\mathbf Z$ (the additive group of integers with the discrete topology; in this case one speaks of a (topological) cascade) and $G=\mathbf R$ (in this case one speaks of a (topological) flow). In the narrow sense, topological dynamical systems refer to these two cases. Sometimes $G$ is not a group but a semi-group. Basically, however, one considers only the semi-group of non-negative integers (in other words, one considers iteration of some continuous mapping $T\colon W\to W$) or (more rarely) of non-negative real numbers.
The term "topological dynamical system" (usually without the first adjective) belongs to topological dynamics, while in topology the same object is called a continuous transformation group. The different terminologies are partly due to the fact that the two disciplines study different properties of the object, and impose different restrictions on it. Thus, a lot of results in topology concern a compact group $G$, whereas in topological dynamics $G$ is usually taken to be locally compact, but never compact, and the interest is in the limiting behaviour of the trajectory $F(g,w)$ as $g\to\infty$ (that is, outside arbitrarily large compact parts of $G$), which even in the analytic case can be extremely complicated. In the theory of algebraic transformation groups (cf. Algebraic group of transformations) one does not assume compactness of $G$, but on the other hand there is a very strong condition of regularity for $F$ as a mapping between algebraic varieties (where the ground field is usually assumed to be algebraically closed, so that in the classical case one is talking about regularity "in the entire complex domain"). Combined with connectivity (and usually also reductivity) of $G$, this enables one, as in the compact case, to obtain significant information about the possible types of mutually adjoining orbits (cf. Orbit), and, in particular, to exclude various phenomena associated with complicated limiting behaviour of trajectories.
The term "topological dynamical system" should be preferred to the commonly used "continuous dynamical system (a flow, a cascade)", because "continuity" can mean also: a) metric continuity, cf. Continuous flow 1); and b) $G=\mathbf R$. (When a topological dynamical system is taken in the narrow sense, one says that a flow is the case of continuous time, and a cascade is the case of discrete time. One sometimes speaks of a continuous and a discrete dynamical system, respectively.)
For references cf. also Topological dynamics.
[a1] | J. de Vries, "Topological transformation groups" , I , Math. Centre (1975) Zbl 0315.54002 Zbl 0299.54030 |
[a2] | R. Ellis, "Lectures on topological dynamics" , Benjamin (1969) MR0267561 Zbl 0193.51502 |
[a3] | I.U. Bronshtein, "Extensions of minimal transformation groups" , Sijthoff & Noordhoff (1979) (Translated from Russian) MR0550605 |