Anima Anandkumar is a prominent researcher about tensor methods (a.k.a., spectral methods) in machine learning. Recently, she has a QA session on Quora and I gathered some of her answers particular with tensor methods as follows:

1. What are some benefits and drawbacks of using tensor methods as opposed to more traditional techniques in machine learning?
   The main gain is in terms of computation:
   — a) tensor methods are embarrassingly parallel and scalable to  large problems
   — b) they can build on efficient linear algebraic libraries, but are much more powerful and informative compared to matrix methods.
   On the other hand, tensor methods are not sample efficient, meaning they require more samples than EM to reach the same level of accuracy (assuming computation is not an issue). Improving statistical efficiency of spectral methods is an ongoing research topic
2. What are your thoughts on the statistical efficiency of spectral methods? Do you think that they are competitive as they stand?
   The short answer is that, MLE is sample efficient but may be difficult to compute while tensor methods (moment matching) is relatively easy to compute but sample inefficient. Some remedies are mentioned in the answer.
3. How are Tensor methods used in deep learning?
   The short answer is that, currently used limited.
4. What are the best resources for starting with Tensor Analysis?
   See her webpage for a start.

For detailed QA, please refer to Quora website.