@codemojo 10 July 2020 more from that month

Catching spammy comments with Markov chains

I read @jtoomim 's ASERT as the new DAA upgrade proposal and though I'm not tech savvy enough to comment on the proposal itself, I did find it extremely annoying that there were loads of comments on that article, which can be - by almost any measure - classified as spam. https://read.cash/@jtoomim/bch-upgrade-proposal-use-asert-as-the-new-daa-8887b9c1 As I was thinking on how one could try to solve this annoyance, I remembered the good ol' Markov chains.

Here's how it works. Given a list of sample comments, for each comment *C*: manually assign score *S* to *C* extract words from *C* update the chain with word *W1* pointing to the next word, *W2*, in the sentence, and this connection between *W1* and *W2* holds the score *S12* + *S*, where *S12* is the score that might have been assigned to this pair of words by a previous comment To clarify a bit, here are the scores for the very first word in a comment: In [1]: m.chain[None] Out[1]: {'hi': 1, 'wow': -4, 'thanks': -4, 'such': -1, 'thank': -2, [...] } So, if a comment starts with the word "hi", it adds **1** to its total score, but if it starts with "wow", the score is reduced by **4**. By now you can guess that this "anti-spam" system can easily be defeated... well, yes but I'm assuming a casual spammer won't have the know-how on how to bypass the trap. So, we built the chain with the comments from the mentioned article. Now, let's try to evaluate some made up comments. In [2]: m.evaluate_words(['exponential', 'moving', 'average', 'ema', 'sounds', 'like', 'a', 'decent', 'solution']) None --[ -0.6 ]--> exponential exponential --[ 1.0 ]--> moving average --[ 1.0 ]--> ema ema --[ 1.0 ]--> sounds like --[ 1 ]--> a a --[ 0.8 ]--> decent Out[2]: 4.2 If there's no connection between *W1* and *W2*, we take the average score of all the pairs having *W1* as the first word. So if "like" points to "a" and "this", and we have to evaluate the pair ("like" --[ 0.5 ]--> "some"), the score will be the average of ("like" --[ 1 ]--> "a") and ("like" --[ 0 ]--> "this" ). In [3]: m.evaluate_words(['thanks', 'exponential', 'moving', 'average', 'sounds', 'like', 'a', 'cool', 'solution', 'thanks', 'for', 'sharing']) None --[ -4 ]--> thanks thanks --[ -1.5 ]--> exponential exponential --[ 1.0 ]--> moving average --[ 1.0 ]--> sounds like --[ 1 ]--> a a --[ 0.8 ]--> cool cool --[ 1.0 ]--> solution solution --[ 1.0 ]--> thanks thanks --[ -4 ]--> for for --[ -7 ]--> sharing Out[3]: -10.7 In [4]: m.evaluate_words(['dude', 'this', 'was', 'hard', 'to', 'follow']) Out[4]: 3.7 In [5]: m.evaluate_words(['bch', 'price', 'will', 'soar', 'good', 'luck']) Out[5]: -4.7 In [6]: m.evaluate_words(['i', 'don', 't', 'know', 'much', 'about', 'the', 'technical', 'side', 'but', 'this', 'was', 'informative']) Out[6]: 12.7 In [7]: m.evaluate_words(['good', 'luck', 'with', 'your', 'endeavour', 'sir']) Out[7]: -4.8 In [8]: m.evaluate_words(['good', 'article', 'shows', 'obviously', 'bch', 'will', 'be', 'crucial', 'without', 'daa']) Out[8]: -4.1 In [9]: m.evaluate_words(['variations', 'of', 'what', 'do', 'you', 'mean', 'change', 'in', 'block', 'interval', 'or', 'hashrate']) Out[9]: 17.3 In [10]: m.evaluate_words(['the', 'problem', 'is', 'not', 'one', 'and', 'exact']) Out[10]: 14.9 The complete python code is here: markov1.py gist https://gist.github.com/jurev/d1fd3f27087e30677b96c4bd0857b12e