The analog of a dot product of vectors is an integral over the product of functions.

The matrix multiplication of vectors - or a row and a column vector - which is then just taking the dot product is called an inner product. So for functions the inner product is an integral over where the functions are defined -

< f, g> = \int f(x) g(x) dx

Likewise you can multiply functions by a "kernel" which is a bit like multiplying a vector by a matrix

< A f, g> = \int \int A(x,y) f(y) g(x) dx dy

The fourier transform is a particular kernel