Although I’m late to the start, I’ve been working on the Kaggle Toxic Comment Challenge. The dataset only contains about 560K comments. Before trying a deep learning model, I was curious to see how well a relatively simple approach might do. At the same time, I still wanted to use word embeddings to maximize generalization to unseen text.

The Coursera Deep Learning Sequence Models class described summarizing documents by averaging over each word’s embedding vector. Some quick research led me to this paper that, in part, describes using the TF-IDF weighted sum of embedding vectors instead. That seemed cool so I decided to give it a try.

I’ll describe how I did this using a subset of the
20 newsgroups dataset. The
scikit-learn package provides a helper function
to do this (note that you may want to change the `data_home`

parameter).

```
from sklearn.datasets import fetch_20newsgroups
news_train = fetch_20newsgroups(subset='train', categories=['sci.space','rec.autos'], data_home='~/local/tmp')
```

This data set has 1187 documents, 594 from `rec.autos`

and 593 from `sci.space`

. Next, we’ll load
the spaCy Python package and its medium-size
English language model which includes pre-computed
Glove embedding vectors. See the spaCy page for instructions
on downloading the language model.

```
import spacy
nlp = spacy.load('en_core_web_md')
```

With spaCy loaded, the newsgroup documents can be lemmatised.

```
def keep_token(t):
return (t.is_alpha and
not (t.is_space or t.is_punct or
t.is_stop or t.like_num))
def lemmatize_doc(doc):
return [ t.lemma_ for t in doc if keep_token(t)]
docs = [lemmatize_doc(nlp(doc)) for doc in news_train.data]
```

Next, the Gensim package can be used to create a dictionary and filter out stop and infrequent words (lemmas).

```
from gensim.corpora import Dictionary
from gensim.models.tfidfmodel import TfidfModel
from gensim.matutils import sparse2full
docs_dict = Dictionary(docs)
docs_dict.filter_extremes(no_below=20, no_above=0.2)
docs_dict.compactify()
```

This process yields a vocabulary with 1191 words. Gensim can again be used to create a bag-of-words representation of each document, build the TF-IDF model, and compute the TF-IDF vector for each document.

```
import numpy as np
docs_corpus = [docs_dict.doc2bow(doc) for doc in docs]
model_tfidf = TfidfModel(docs_corpus, id2word=docs_dict)
docs_tfidf = model_tfidf[docs_corpus]
docs_vecs = np.vstack([sparse2full(c, len(docs_dict)) for c in docs_tfidf])
```

The result, `docs_vecs`

, is a matrix with 1187 rows (docs) and 1191 columns
(TF-IDF terms). Next, we use spaCy to get the 300 dimensional Glove embedding vector for
each TF-IDF term.

```
tfidf_emb_vecs = np.vstack([nlp(docs_dict[i]).vector for i in range(len(docs_dict))])
```

The `tfidf_emb_vecs`

matrix has 1192 rows (terms) and 300 columns (Glove vectors). To
get a TF-IDF weighted Glove vector summary of each document, we just need to
matrix multiply `docs_vecs`

with `tfidf_emb_vecs`

.

```
docs_emb = np.dot(docs_vecs, tfidf_emb_vecs)
```

As expected, `docs_emb`

is a matrix with 1187 rows (docs) and 300
columns (Glove vectors).

To get sense of how well these document summaries do, we can use PCA to reduce the dimensionality,

```
from sklearn.decomposition import PCA
docs_pca = PCA(n_components=8).fit_transform(docs_emb)
```

and then use t-sne to project the vectors to 2D.

```
from sklearn import manifold
tsne = manifold.TSNE()
viz = tsne.fit_transform(docs_pca)
```

Plotting with matplotlib…

```
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.margins(0.05)
zero_indices = np.where(news_train.target == 0)[0]
one_indices = np.where(news_train.target == 1)[0]
ax.plot(viz[zero_indices,0], viz[zero_indices,1], marker='o', linestyle='',
ms=8, alpha=0.3, label=news_train.target_names[0])
ax.plot(viz[one_indices,0], viz[one_indices,1], marker='o', linestyle='',
ms=8, alpha=0.3, label=news_train.target_names[1])
ax.legend()
plt.show()
```

Produces the following plot which shows that the TF-IDF weighted Glove vectors did a decent job of separating the documents.

I did some additional feature engineering to further improve separability for the Kaggle competition. After fitting an SVM binary classification model for each label, I was able to achieve a score of 0.9470 on the mean column-wise ROC AUC metric used by the competition. While not bad, it’s far from the best leaderboard scores. I may experiment some more with weighted embedding vectors but I fear too much information is lost in the summarizing to get to the top of the board.