Jekyll2021-06-10T14:09:33+00:00https://ronaldluc.com/feed.xmlRonald LucMachine learning, teaching, life.Ronald LucRetail stock market bubble simulation2021-06-04T06:47:00+00:002021-06-04T06:47:00+00:00https://ronaldluc.com/ml/system-models<p>In collaboration with <a href="493127@muni">Tomáš Macháček</a></p>
<p><a href="https://github.com/ronaldluc/retail-bubble-stock-market-simulation">Code: github.com/ronaldluc/retail-bubble-stock-market-simulation</a></p>
<p><a href="https://colab.research.google.com/drive/1BscDMEW5yGmxJyO1lsfXYtUJDRXFKxzy?usp=sharing">Google Collab: 🤮 url</a></p>
<h1 id="stock-market-simulation">Stock market simulation</h1>
<h2 id="introduction">Introduction</h2>
<p>In the past few months, the mania has taken over the world markets. Many countries have decided to use monetary expansion to battle the economic burdens of COVID‒19. This rapid increase of money printing has brought unusual amounts of money, especially to US households, resulting in quick asset price increases. First, the wave has taken over normal and rather safer assets, like Fortune 500 companies. After some time, the greediness of investors rose, and so did the proportion of money flowing into risky assets. As a result, we have seen rises in cryptocurrency and prices of small companies. In the end, many retail investors stopped investing and started participating in extremely risky strategies. This kind of behavior is present in the markets most of the time. The difference that made us take an interest in this was that usually, when these kinds of events happen, they tend to die out quickly, and there aren’t multiple assets getting “pumped up” at once. This time there were multiple assets over multiple markets with the same behavior, and the whole scheme did not stop for a couple of months.</p>
<p><strong>Definitions</strong></p>
<ul>
<li>Stock (Equity) — security that represents the ownership of a fraction of a corporation</li>
<li>Investor — real-world entity buying equities</li>
<li>Retail investor — individual, non-professional investor</li>
<li>Momentum — velocity of stock price change over time period</li>
<li>Price action — the price of the stock over time</li>
<li>Pump and dump — is a scheme that attempts to boost the price of a stock through recommendations based on false, misleading, or greatly exaggerated statements</li>
</ul>
<h2 id="problem-formulation">Problem formulation</h2>
<p>How the overall strategies of investors and market parameters change the price action of stock during pump and dump events.
Gains and losses with respect to the investment strategy.
What seems to be the best investment strategy in these types of environments.</p>
<figure>
<a href="../../images/2021/06/GME.png"><img src="../../images/2021/06/GME.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/GNUS.png"><img src="../../images/2021/06/GNUS.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/HTZQ.png"><img src="../../images/2021/06/HTZQ.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<h2 id="real-life-examples">Real-life examples</h2>
<p>In the real markets, pump and dump events are common occurrences. They usually occur on small, mostly worthless stocks. Usually, a small group of investors buy at a very low price and convince more investors to get in, with a vision of huge profit. This kind of thinking spreads like an epidemic since all of these new investors convince some of their friends to invest. This kind of behavior continues until most of the people willing to invest in this kind of scheme get in or until the hype dies out and people get bored, hence sell everything.</p>
<!-- #### Shorting
We could separate them into many categories, but for simplicity, we will focus on short-squeezes because recent events were mostly short-squeezes. In the real world, there are multiple strategies on how to make money in the markets; one of them is called shorting the stock. Shorting means borrowing shares and selling them to whom you have borrowed them from and then buying them back at a different price and returning them. This type of strategy is used when you expect the stock price to plummet. On rare occasions, for example, now some stocks get shorted for obnoxious %, even above 100%. This is a perfect opportunity for a pump and dump scheme to arise. The idea is that person who is shorting can lose more than 100% of their investment even to the point where they have to liquidate their other assets to cover for one short. People running the scheme buy in at very low prices and spread the hype; this causes shorts to cover, which causes the price to go up, which causes more shorts to cover… At a certain point, people running the scheme sell everything and cause the price to drop, or people start losing interest. -->
<h2 id="model-definition">Model definition</h2>
<p>In order to simulate the real market, we had to first create our artificial stock market. The price of the stock is simulated by an exponential curve with a very small base. Each tick of the model, investors may choose to buy or sell.</p>
<p><strong>To buy</strong></p>
<ol>
<li>At the start of each tick, the amount of money to be traded is calculated.</li>
<li>Using a geometric sum, the total amount of shares to be bought is calculated.</li>
<li>The shares are distributed between investors in proportion to the money spent.</li>
<li>The new price per share is set, and the transaction is completed.</li>
</ol>
<p>Likewise, to sell stocks.</p>
<p><strong>For such stock market simulation holds</strong></p>
<ul>
<li>All investors selling all stocks at once results in everyone getting a proportion of the current market cap.</li>
<li>If the same investor solo-buys in one tick and solo-sells in the next, nothing changes.</li>
<li>More generally, if investors sell in the reverse order to how they bought, everyone ends up with the same amount of money.</li>
</ul>
<p>In the real stock market, price calculation works differently and depends on many variables that we are not considering in this model, such as share structure, spreads, and volatility.</p>
<p>We have decided to use an agent-based model for investors. Every investor starts with basic parameters that affect their trading decisions. In the beginning, there are only a couple of investors that are aware of the stock. Each tick, they have a small chance of telling someone else about the stock, hence spreading the hype. This so-called hype epidemic quickly spreads through all potential investors and causes the stock price to skyrocket since most of them wouldn’t want to miss on this investment opportunity. On the other hand, these investors are not stupid, and when the price is rising too quickly they have a chance to sell their shares. These decisions to sell and buy are based on investor behavioral function. The final factor contributing to our model functionality is boredom. After the scheme has been active for some time, some investors get bored and sell everything they have.</p>
<h3 id="behavior-function">Behavior function</h3>
<p>Behavior function is our way of representing the internal thoughts of an investor in a stock market. Every investor bases his decisions on his own parameters and the current stock price change, stock price delta. When the price is trending up or down slowly, most of the investors go along with the momentum; however, if the price starts rising or falling too quickly, some of the investors start thinking smart and do the opposite of what the rest of the investors are doing.</p>
<p>Input is <code class="language-plaintext highlighter-rouge">delta_price</code> (x-axis), ouput is buy/sell probability (y-axis). Best viewed with grid on full screen.
<!-- <iframe src="https://www.geogebra.org/geometry/eehs5tpw?embed" width="1200" height="600" allowfullscreen style="border: 1px solid #e4e4e4;border-radius: 4px;" frameborder="0"></iframe> --></p>
<!-- <div style="overflow: hidden">
<iframe src="https://www.geogebra.org/geometry/eehs5tpw?embed" allowfullscreen style="border: 1px solid #e4e4e4; border-radius: 4px; width: 150%; height: 900px;transform: scale(0.75) translate(-75%, -75%); margin-bottom: -450px;" frameborder="0"></iframe>
</div> -->
<div style="overflow: hidden">
<iframe src="https://www.geogebra.org/geometry/eehs5tpw?embed" allowfullscreen="" style="border: 1px solid #e4e4e4; border-radius: 4px; width: 134%; height: 720px;transform: scale(0.8) translate(-20%, -10%); margin-bottom: -140px;" frameborder="0"></iframe>
</div>
<h3 id="hyperparameters">Hyperparameters</h3>
<ul>
<li>“_exp” in name means the value is searched in \(10^x\) space</li>
<li><em>name (range for search/substitution study): description</em></li>
<li>ranges are chosen to be the wides reasonable (e.g., numerical stability in behavioral function)</li>
</ul>
<p>Behavior hyperparameters</p>
<ul>
<li>mean_break_point_exp (-2, 0): mean of <code class="language-plaintext highlighter-rouge">break</code> distribution</li>
<li>std_break_point_exp (-3, 0): std of <code class="language-plaintext highlighter-rouge">break</code> distribution</li>
<li>std_optimism_exp (-3, 0): std of <code class="language-plaintext highlighter-rouge">optimism</code> distribution</li>
<li>behavior_scale (1, 50): <code class="language-plaintext highlighter-rouge">scale</code></li>
<li>behavior_linearity_exp (-1, 2): <code class="language-plaintext highlighter-rouge">linearity</code></li>
</ul>
<p>Market hyperparameters</p>
<ul>
<li>past_impact_weight (1, 80): weight of history in <code class="language-plaintext highlighter-rouge">delta_price</code> calculation</li>
<li>free_money_multiple (0.1, 100): how many times more money should all investors together have compared to the initial market cap</li>
<li>stock_buy_extra_exp (-6, -2): additional price of the next share compared to the previous one ( \(share_{(n+1)} = share_n * (1 + stock\_buy\_extra )\) )</li>
<li>transmission_prob_exp (-2, -0.9): a chance for each aware agent to spread the knowledge about to any other agent per tick</li>
<li>boredom_prob_exp (-4, -2): a chance for each aware agent to get bored, sell all shares and never look back</li>
<li>delta_price_exp (-6, 0): initial <code class="language-plaintext highlighter-rouge">delta_price</code></li>
</ul>
<h2 id="feedback-loop-diagram">Feedback loop diagram</h2>
<!-- Similar to https://www.fi.muni.cz/~xpelanek/IV109/slidy/zpetna-vazba.pdf
Created here: Flowchart Maker & Online Diagram Software -->
<p>The backbone is a SIR model making an exponentially growing amount of people aware of the stock at the beginning</p>
<ul>
<li>S ~ unaware of the stock</li>
<li>I ~ aware of the stock (and interested in investing)</li>
<li>R ~ bored of the stock</li>
</ul>
<figure>
<a href="../../images/2021/06/feedback_loop.png"><img src="../../images/2021/06/feedback_loop.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<h1 id="evaluation">Evaluation</h1>
<h2 id="a-single-run">A single run</h2>
<p>All values are normalized to <-1, 1> to be shown in one diagram.</p>
<figure>
<a href="../../images/2021/06/single_run.png"><img src="../../images/2021/06/single_run.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/single_run_labeled2.png"><img src="../../images/2021/06/single_run_labeled2.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<h2 id="selecting-baseline">Selecting baseline</h2>
<p>We want a set of hyperparameters that lead to models behavior similar to the one seen in reality. After analyzing a few pumps and dump stock price examples from reality, 4 peaks were chosen with ratios 2:4:18:9 and in-between drops with ratios 1:2:5. Those ratios were used to create a loss, for weights and implementation details see the ipython notebook.</p>
<p>GNUS pump and dump peak ratios examination</p>
<figure>
<a href="../../images/2021/06/GNUS_peaks.png"><img src="../../images/2021/06/GNUS_peaks.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<p>A hyperparameter search using <a href="https://github.com/fmfn/BayesianOptimization">bayes optimization</a> with 3700 steps was used. The simulation is stochastic; therefore, each step is the average loss over 7 runs. The #1 in search is selected as the baseline.</p>
<figure>
<a href="../../images/2021/06/hyper_search_results.png"><img src="../../images/2021/06/hyper_search_results.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<p>The models seems to be unable to perform 2 peaks before the main one. Then if there is a pre-peak, it is always around the linear part of <code class="language-plaintext highlighter-rouge">I</code> curve (awareness spread).</p>
<h2 id="abstracting-the-time">Abstracting the time</h2>
<p>We are looking for the leverage points and phase transitions in the model. To achieve this, we create multiple market and agent markers. Each marker is a single number that represents a property over one market simulation.</p>
<p>Market markers</p>
<ul>
<li>first_peak [ticks]: tick of the first peak</li>
<li>mean_peak [ticks]: mean tick of a peak</li>
<li>last_peak [ticks]: tick of the last peak</li>
<li>peaks_num [#]: number of peaks per simulation</li>
<li>max_market_penetration [fraction]: the maximum fraction of money put into the stock at one time</li>
</ul>
<p>Agent markers</p>
<ul>
<li>max_gain [money]: the most a single agent gained at the end of the simulation</li>
<li>investore_lost_proportion [fraction]: the fraction of agents to end with less money than they started with</li>
<li>money:optimism correlation [corr]: pearson correlation of the gained money to the optimism of agents</li>
<li>money:initial_money correlation [corr]: pearson correlation of the gained money to initial_money of agents</li>
<li>money:break_point correlation [corr]: pearson correlation of the gained money to break_point of agents</li>
</ul>
<p>Each hyperparameter is evaluated on 1000 evenly spread values of a hyperparameter (x-axis). Shown is mean +- 2× std from rolling window of 50. Beware of relative y-axes.</p>
<figure>
<a href="../../images/2021/06/graphs_explanation.png"><img src="../../images/2021/06/graphs_explanation.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<h2 id="results">Results</h2>
<h3 id="the-most-influential-params">The most influential params</h3>
<figure>
<a href="../../images/2021/06/past_impact_weight.png"><img src="../../images/2021/06/past_impact_weight.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/std_optimism_exp.png"><img src="../../images/2021/06/std_optimism_exp.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<p>Changing <code class="language-plaintext highlighter-rouge">past_impact_weight</code> or <code class="language-plaintext highlighter-rouge">std_optimism_exp</code> makes the market penetration and money:optimism corelation undergo similar phase change. Worth noting is also the stable market behavior for any std of optimism over \(10^{-1} = 0.1\) (the optimism is always a normal distribution with 0 mean, so the change is done by a higher diversity in the optimism between people) with 1‒2 peaks.</p>
<h3 id="less-influential-params">Less influential params</h3>
<figure>
<a href="../../images/2021/06/stock_buy_extra_exp.png"><img src="../../images/2021/06/stock_buy_extra_exp.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/behavior_linearoty_exp.png"><img src="../../images/2021/06/behavior_linearoty_exp.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/free_money_multiple.png"><img src="../../images/2021/06/free_money_multiple.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<p>Changing <code class="language-plaintext highlighter-rouge">free_money_multiple</code>, <code class="language-plaintext highlighter-rouge">behavior_linearity_exp</code>, or <code class="language-plaintext highlighter-rouge">stock_buy_extra_exp</code> does not impact the market behavior much for most of the values used in hyperparameter search. If the <code class="language-plaintext highlighter-rouge">free_money_multiple</code> or <code class="language-plaintext highlighter-rouge">stock_buy_extra_exp</code> is small, agents cannot change the stock price too much \(\Rightarrow\) agents cannot gain/lose too much, and the money:<code class="language-plaintext highlighter-rouge">initial_money</code> correlation is high. Higher <code class="language-plaintext highlighter-rouge">behavior_linearoty_exp</code> increases the chance to trade \(\Rightarrow\) from a certain eagerness to trade the market behavior does not change much.</p>
<h3 id="sir-params">SIR params</h3>
<figure>
<a href="../../images/2021/06/transmission_prob_exp.png"><img src="../../images/2021/06/transmission_prob_exp.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/boredom_prob_exp.png"><img src="../../images/2021/06/boredom_prob_exp.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<p>If the <code class="language-plaintext highlighter-rouge">transmission_prob_exp</code>, <code class="language-plaintext highlighter-rouge">boredom_prob_exp</code> are close together, the model oscillates a lot (hundreds of peaks). If the <code class="language-plaintext highlighter-rouge">boredom_prob_exp</code> is too low, not many people buy the stock, but the ones who trade are completely stripped of money by just a few agents. Changing <code class="language-plaintext highlighter-rouge">transmission_prob_exp</code> causes a phase change on money:optimism correlation \(\Rightarrow\) In a slowly spreading bubble, it is worth being a pessimist, and there are plenty of speculations (high # peaks), and most susceptible agents get involved. With quickly spreading hype, it is good to be slightly optimistic, and there are the same ~5 peaks before the meme stock is dead, with fewer people having time to get involved.</p>
<h3 id="break-point">Break point</h3>
<figure>
<a href="../../images/2021/06/mean_break_point.png"><img src="../../images/2021/06/mean_break_point.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/std_break_point_exp.png"><img src="../../images/2021/06/std_break_point_exp.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<p>Based on the graphs shown, it might have been a good idea to search over an even wider range of <code class="language-plaintext highlighter-rouge">break_point</code> normal distributions. This was not done as <code class="language-plaintext highlighter-rouge">break_point == 1</code> means a change in behavior only for <code class="language-plaintext highlighter-rouge">delta_price > 1</code>. With the search done, diversity in <code class="language-plaintext highlighter-rouge">break_point</code> is more important than the mean value.</p>
<h3 id="no-effect">No effect</h3>
<figure>
<a href="../../images/2021/06/delta_price_exp.png"><img src="../../images/2021/06/delta_price_exp.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2021/06/behavior_scale.png"><img src="../../images/2021/06/behavior_scale.png" /></a>
<figcaption> <br />
Source: own work</figcaption>
</figure>
<p><code class="language-plaintext highlighter-rouge">delta_price_exp</code> and <code class="language-plaintext highlighter-rouge">behavior_scale</code> show little impact on the model behavior.</p>
<h1 id="conclusions">Conclusions</h1>
<p>Overall, we were able to create a solid model that, with our level of abstraction, answered the questions we set out to answer at the beginning.</p>
<h2 id="general-questions-about-the-market">General questions about the market</h2>
<p>Our first goal was to determine what are the most critical factors for the whole bubble to arise in the first place.
Looking back at <code class="language-plaintext highlighter-rouge">past_impact_weight</code> and <code class="language-plaintext highlighter-rouge">std_optimism_exp</code>, it becomes clear that these kinds of bubbles occur when people use the stock market as a casino. The two critical conditions for the bubble to arise seem to be the rash decisions, represented by low <code class="language-plaintext highlighter-rouge">past_impact_weight</code> where one looks only at low timeframes and herd mentality, represented by small deviations from base <code class="language-plaintext highlighter-rouge">optimism</code>. Even small changes in these parameters cause investors to act as if there was no bubble and the market was acting rationally. We can mostly see it in the <code class="language-plaintext highlighter-rouge">past_impact_weight</code> parameter, where even mid-ranged time frames simulate normal asset behavior. The best strategy in this type of market is to buy in and hold regardless of short-term events and price changes. We can see the same stabilizing effects with higher <code class="language-plaintext highlighter-rouge">std_optimism_exp</code>. In this case, it is less obvious why this stabilization happens. The point seems to be that more significant differences in personalities cause investors to act more rationally overall. When the stock goes up too quickly, pessimistic investors hold it down, so it can’t turn into a bubble, and optimistic investors buy even small dips, so the price cannot drop too quickly.</p>
<p>Our next goal was to determine which parameters contribute the most to investors losing money in these events.
Looking at our model and the real world, the basic answer to this question is that most people lose money because they risk. As we can see, the most influential parameters concerning the overall percentage of people who lost their money are the ones critical for a bubble to arise. From this, we conclude that participating in risky assets is what makes most people lose money.
Our model provides even more insight into this problem when we look at <code class="language-plaintext highlighter-rouge">free_money_multiple'and </code>stock_buy_extra_exp<code class="language-plaintext highlighter-rouge">. These two parameters represent the two reasons for stock volatility. Higher </code>free_money_multiple<code class="language-plaintext highlighter-rouge"> gives people more money, so they can get a bigger part of the company early on. This negatively affects overall investors because when these "whales" eventually sell their shares, they cause the market to crash on their own, hence hurt a larger amount of people in the process. </code>stock_buy_extra_exp` has similar effects. In this case, the first investors get in at a low price and get a big share of the company, while the next investors get a disproportionally lower share of the company for the same amount. This again means that when the first investors sell, the crash inevitably happens.</p>
<h2 id="question-about-investor-behavior">Question about investor behavior</h2>
<p>Our last goal was to look and try to figure out good strategies in bubble stocks.
We can’t fully answer this question using our simple model, but we can look at some interesting readings to try to determine the best behavior. First, looking at the critical conditions for the bubble to arise, we can see that optimism and cautiousness represented by low <code class="language-plaintext highlighter-rouge">break_point</code> positively predict success in the bubble market.
Next, looking at <code class="language-plaintext highlighter-rouge">free_money_multiple</code> and <code class="language-plaintext highlighter-rouge">stock_buy_extra_exp</code>, we can see that being optimistic in a low market cap environment is proficient.</p>
<h2 id="possible-extensions">Possible extensions</h2>
<p>General market</p>
<ul>
<li>Price spread model</li>
<li>Share structure model</li>
<li>Model epidemic outbursts (stock mentioned in the news)</li>
</ul>
<p>Investor behavior</p>
<ul>
<li>Add wealthy investors</li>
<li>Change every parameter into a distribution sample for each agent</li>
<li>Add more parameters into the behavioral function</li>
</ul>
<h2 id="tldr">TLDR</h2>
<p>Taking into consideration what we had found out, <strong>the single best strategy is to never invest in assets with this behavior</strong>.</p>Ronald LucWhat behavioral traits are important to :win: and what impacts the overall market behaviorMy first big talk2019-11-29T06:47:00+00:002019-11-29T06:47:00+00:00https://ronaldluc.com/events/we-are-developers<!-- Courtesy of embedresponsively.com //-->
<div class="responsive-video-container">
<iframe src="https://www.youtube.com/embed/tWSVNgxcw5s" frameborder="0" allowfullscreen=""></iframe>
</div>
<p>My colleague and I had Sanchit Singh had the honor to present at WeAreDevelopers Congress Vienna 2019. We wanted to support the whole talk with (ideally) real-world examples. We decided to show most of the ideas on exaggerated use cases in <a href="https://colab.research.google.com/notebooks/welcome.ipynb">Google Collab</a> with training right on the stage. <a href="http://bit.ly/2qWpvts">The code can be found here</a>.</p>
<figure>
<a href="../../images/2019/11/audience.jpg"><img src="../../images/2019/11/audience.jpg" /></a>
<figcaption>Our stage. Source: own work</figcaption>
</figure>
<p>The video of the talk is not yet published, so for now following is a small summary of what we presented.</p>
<h2 id="why-use-transfer-learning-tr">Why use Transfer learning? (TR)</h2>
<ul>
<li>To reuse a big model (trained once on millions of samples)</li>
<li>To use Deep Learning even on normally too small datasets</li>
<li>To increase accuracy by leveraging the extra information from different data</li>
</ul>
<p>In this article, we will work with ResNet50<sup id="fnref:fa" role="doc-noteref"><a href="#fn:fa" class="footnote" rel="footnote">1</a></sup> architecture pre-trained on the ImageNet<sup id="fnref:fb" role="doc-noteref"><a href="#fn:fb" class="footnote" rel="footnote">2</a></sup> dataset. All code will be using <a href="https://docs.fast.ai/">fastai</a> library, which is included in the Collab environment.</p>
<figure>
<a href="../../images/2019/11/imagenet.png"><img src="../../images/2019/11/imagenet.png" /></a>
<figcaption>Example images from ImageNet.<br />Source: <a href="http://www.image-net.org/">ImageNet</a></figcaption>
</figure>
<p>Code to create ResNet50 with pretrained weights on ImageNet:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_resnet</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="o">-></span> <span class="n">Model</span><span class="p">:</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">resnet50</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">cnn_learner</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">metrics</span><span class="o">=</span><span class="p">[</span><span class="n">accuracy</span><span class="p">])</span>
<span class="k">return</span> <span class="n">model</span>
</code></pre></div></div>
<h3 id="classifying-cats-and-dogs">Classifying cats and dogs</h3>
<p>The most common example of them all. Train set has 363 images, validation 90 images and test 463 images. The large size of the test set compared to validation or even train is really important for such a small data. Otherwise we may see totally different results in the inference. Let’s perform one cycle using the one cycle policy<sup id="fnref:fc" role="doc-noteref"><a href="#fn:fc" class="footnote" rel="footnote">3</a></sup>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">train</span><span class="p">(</span><span class="n">model</span><span class="p">:</span> <span class="n">Model</span><span class="p">,</span> <span class="n">cyc_len</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">-></span> <span class="n">Model</span><span class="p">:</span>
<span class="n">callbacks</span> <span class="o">=</span> <span class="p">[</span><span class="n">EarlyStoppingCallback</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">monitor</span><span class="o">=</span><span class="s">'valid_loss'</span><span class="p">,</span> <span class="n">min_delta</span><span class="o">=</span><span class="mf">0.01</span><span class="p">,</span> <span class="n">patience</span><span class="o">=</span><span class="mi">2</span><span class="p">)]</span>
<span class="n">model</span><span class="p">.</span><span class="n">fit_one_cycle</span><span class="p">(</span><span class="n">cyc_len</span><span class="o">=</span><span class="n">cyc_len</span><span class="p">,</span> <span class="n">max_lr</span><span class="o">=</span><span class="mf">1e-3</span><span class="p">,</span> <span class="n">callbacks</span><span class="o">=</span><span class="n">callbacks</span><span class="p">)</span>
<span class="k">return</span> <span class="n">model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">get_resnet</span><span class="p">(</span><span class="n">train_data</span><span class="p">)</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">train</span><span class="p">(</span><span class="n">model</span><span class="p">)</span>
</code></pre></div></div>
<p>For better evaluation, we will use the confusion matrix of the validation and test set.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">eval</span><span class="p">(</span><span class="n">model</span><span class="p">:</span> <span class="n">Model</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">''</span><span class="p">):</span>
<span class="n">model_copy</span> <span class="o">=</span> <span class="n">deepcopy</span><span class="p">(</span><span class="n">model</span><span class="p">)</span>
<span class="n">model_copy</span><span class="p">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">data</span>
<span class="n">_</span><span class="p">,</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">model_copy</span><span class="p">.</span><span class="n">validate</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">valid_dl</span><span class="p">,</span> <span class="n">metrics</span><span class="o">=</span><span class="p">[</span><span class="n">accuracy</span><span class="p">])</span>
<span class="n">interp</span> <span class="o">=</span> <span class="n">ClassificationInterpretation</span><span class="p">.</span><span class="n">from_learner</span><span class="p">(</span><span class="n">model_copy</span><span class="p">)</span>
<span class="n">interp</span><span class="p">.</span><span class="n">plot_confusion_matrix</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">12</span><span class="p">,</span><span class="mi">12</span><span class="p">),</span> <span class="n">dpi</span><span class="o">=</span><span class="mi">60</span><span class="p">,</span> <span class="n">return_fig</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">display</span><span class="p">(</span><span class="n">HTML</span><span class="p">(</span><span class="sa">f</span><span class="s">'<h1></span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s"> accuracy: </span><span class="si">{</span><span class="nb">float</span><span class="p">(</span><span class="n">acc</span><span class="p">)</span><span class="o">*</span><span class="mi">100</span><span class="p">:</span><span class="mf">3.4</span><span class="si">}</span><span class="s"> %</h1>'</span><span class="p">))</span>
<span class="nb">eval</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">train_data</span><span class="p">,</span> <span class="s">'Val'</span><span class="p">)</span>
</code></pre></div></div>
<figure>
<a href="../../images/2019/11/val1.png"><img src="../../images/2019/11/val1.png" /></a>
<figcaption>Validation accuracy: 98.89 %</figcaption>
</figure>
<p>That is a great result. It was to be expected as the different breeds of cats and dogs are already present in the ImageNet dataset. What is the accuracy of the test set?</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">eval</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">test_data</span><span class="p">,</span> <span class="s">'Test'</span><span class="p">)</span>
</code></pre></div></div>
<figure>
<a href="../../images/2019/11/test1.png"><img src="../../images/2019/11/test1.png" /></a>
<figcaption>Test accuracy: 81.64 %</figcaption>
</figure>
<p>That is significantly worse than the validation. We are overfitting the train set.</p>
<p>Want more? Please watch the capture from the conference, or run <a href="http://bit.ly/2qWpvts">the code from gCollab</a>.</p>
<h2 id="references">References</h2>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:fa" role="doc-endnote">
<p>Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. 2015. Deep Residual Learning for Image Recognition. <a href="https://arxiv.org/abs/1512.03385">arXiv:1512.03385</a> <a href="#fnref:fa" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fb" role="doc-endnote">
<p>Deng, J. and Dong, W. and Socher, R. and Li, L.-J. and Li, K. and Fei-Fei, L. 2009. ImageNet: A Large-Scale Hierarchical Image Database. <a href="http://www.image-net.org/">image-net.org</a> <a href="#fnref:fb" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fc" role="doc-endnote">
<p>Leslie N. Smith. 2017. A disciplined approach to neural network hyper-parameters: Part 1 – learning rate, batch size, momentum, and weight decay. <a href="https://arxiv.org/abs/1803.09820">arXiv:1803.09820</a> <a href="#fnref:fc" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Ronald LucCommon pitfalls in a Computer Vision projectJak vybrat sluchátka2019-09-26T06:47:00+00:002019-09-26T06:47:00+00:00https://ronaldluc.com/random/how-to-select-headphones<p>Informace zde uvedené jsou jen moje náhledy a generalizace. Vše z 15–30 různých sluchátek, které jsem měl doma; zkoušení se sluchátky otestovanými když jdu zrovna do elektra; sledování videí na YT a čtení cizích článků. Ne pravda. Pokud jsou nějaká fakta špatně, kontaktujte mě na <a href="https://www.linkedin.com/in/ronald-luc">LinkedInu</a>.</p>
<h1 id="předpoklady">Předpoklady</h1>
<blockquote>
<p>Chci hlavně poslouchat hudbu a sledovat videa</p>
</blockquote>
<ul>
<li>Na hraní u PC jsou lepší kabelová sluchátka</li>
<li>Mají nižší latenci a cenu za zvuk</li>
</ul>
<blockquote>
<p>Kabelová sluchátka jsou mrtvá.</p>
</blockquote>
<figure>
<a href="../../images/2019/09/wireless.jpeg"><img src="../../images/2019/09/wireless.jpeg" /></a>
<figcaption>Přesto to není důvod nemít v mobilu 3.5 jack! <br />
Source: <a href="https://memedroid.com">memedroid</a></figcaption>
</figure>
<blockquote>
<p>Většina lidí nedocení malé zlepšení v kvalitě zvuku</p>
</blockquote>
<ul>
<li>Stejně posloucháme málobitovou hudbu</li>
<li>Pokud někdo velmi pozná rozdíl mezi <em>dobrými</em> sluchátky za 1 000 a 10 000 Kč, je to audiophil a nepotřebuje pomoc s výběrem, nýbrž pomoc obecně</li>
</ul>
<blockquote>
<p>Správný <em>typ</em> sluchátek s dobrým hodnocením na Heurece => spokojenost</p>
</blockquote>
<ul>
<li>Právě s dobrým výběrem a návodem co vůbec hledat pomůže tento článek*</li>
</ul>
<blockquote>
<p>Jakmile vím, co chci, zvládnu si vybrat</p>
</blockquote>
<p>Postup:</p>
<ol>
<li>Přečtu tento článek</li>
<li>Rozmyslím si co chci</li>
<li>Naklikám to jako filtery do Heureky
<ul>
<li>cenu nechám “nahoře” trochu vyšší ať vím zda o něco nepřicházím</li>
</ul>
</li>
<li>Koupím horních N sluchátek s nejlepšími recenzemi</li>
</ol>
<figure>
<a href="../../images/2019/09/headphones.png"><img src="../../images/2019/09/headphones.png" /></a>
<figcaption>Různé typy sluchátek <br />
Source: <a href="https://bezdratova-sluchatka.heureka.cz">Heureka</a></figcaption>
</figure>
<h1 id="sada-otázek-a-doporučení">Sada otázek a doporučení</h1>
<h2 id="k-čemu-je-budu-používat">K čemu je budu používat?</h2>
<p>Jsou 3 hlavní typy sluchátek</p>
<ul>
<li>Přes hlavu
<ul>
<li>15–30 h výdrže</li>
<li>supr zvuk, dají se dát kolem krku, u sportu se zapotíš</li>
</ul>
</li>
<li>Do uší spojené drátkem
<ul>
<li>6–10 h výdrže</li>
<li>rozumný zvuk, dají se dát kolem krku, supr na sport</li>
</ul>
</li>
<li>Do uší oddělené
<ul>
<li>2–4 h výdrže s krabičkou která sluchátka až 5× nabije během hodiny</li>
<li>musí se vždy uklízet, u sportu spiš vypadnou (existují modely s lalokovou výplní)</li>
</ul>
</li>
</ul>
<h2 id="další-údaje-ke-zvážení">Další údaje ke zvážení</h2>
<ul>
<li>Pot odolnost – zvlášť u těch do uší, aby při sportu nepřestala naráz fungovat</li>
<li>Konstrukce – sluchátka přes hlavu s otočnými <em>plastovými</em> panty moc nevydrží</li>
<li>Skladnost – typ složení/ohebnosti pro “přes hlavu”, velikost nabíjecí krabičky pro špuny</li>
</ul>
<h2 id="jaký-má-můj-mobil-bluetooth">Jaký má můj mobil Bluetooth?</h2>
<p>Koupit si sluchátka se stejným / novějším Bluetooth.
Bluetooth 4.0–5.0 jsou plně vzájemně kompatibilní</p>
<ul>
<li>3.0 – přemýšlet o novém telefonu</li>
<li>4.0 – stále bych přemýšlel</li>
<li>4.2 – nižší spotřeba, obzvlášť v stand-by režimu</li>
<li>5.0 – oboustranně kvalitní zvuk při telefonování, větší dosah v stand-by bez odpojení
<ul>
<li>Předchozí verze při oboustranném toku zvuku změnili protokol na bazmek</li>
</ul>
</li>
</ul>
<h2 id="jaký-mám-rozpočet">Jaký mám rozpočet?</h2>
<p>Údaje aktuální v ČR obchodech k 2019</p>
<ul>
<li>pod 500 Kč – nic moc</li>
<li>500–1 000 Kč – zvlášť ve slevě některé dobré kusy, přesto s kompromisy</li>
<li>1 000–2 000 Kč – zvlášť ve slevě moc dobré kusy
<ul>
<li>Sluchátka přes hlavu má smysl kupovat i trochu přes 2 000 Kč</li>
<li>Aktivnímu rušení hluku bych pod 2 000 Kč nevěřil</li>
</ul>
</li>
<li>2 000–více Kč – platí se spíš za značku, většinou neopodstatnělé – jen většinou!
<ul>
<li>Například Beats by Dr. Dre jsou dost marketing</li>
<li>Pokud někdo <em>potřebuje</em> AirPods, <em>nepotřebuje</em> tento návod ale iPhone</li>
</ul>
</li>
</ul>
<h2 id="objednat-si-jich-několik">Objednat si jich několik</h2>
<h3 id="a-do-14-dní-všechny-bez-jedněch-vrátit">a do 14 dní všechny bez jedněch vrátit</h3>
<ul>
<li>Všichni mají jiné uši</li>
<li>Každému vadí nějaká drobnost (mě například lehké bzučení když jsou sluchátka zapnutá, připojená ale bez zvuku)</li>
<li>Obzvlášť dobré, když si člověk není jistý co chce – objednat více různých typů</li>
</ul>
<figure>
<a href="../../images/2019/09/chineseHeadphones.jpeg"><img src="../../images/2019/09/chineseHeadphones.jpeg" /></a>
<figcaption>Ebay či Alibaba úplně nemají 2 letou záruku <br />
Source: <a href="https://memedroid.com">memedroid</a></figcaption>
</figure>
<h1 id="podrobnější-tipy-k-typům">Podrobnější tipy k typům</h1>
<h2 id="přes-hlavu">Přes hlavu</h2>
<ul>
<li>Uzavřená / otevřená – otevřená více “prosakují zvuk” zvenku (a naopak hudbu ven!)</li>
<li>Na uši / úplně obklopující uši
<ul>
<li>Z uší obklopujících mám za 30 minut divný pocit “tlaku” na bubínku + jsou těžší</li>
<li>Na uši některé lidi příliš tlačí => zkusit týden hodně nosit</li>
</ul>
</li>
<li>Dražší modely s aktivním rušením šumu – zkoušel jsem:
<ul>
<li>Sony, Bose – oboje skutečně neskutečné. Je to jak vypnout okolní svět</li>
<li>Nejlevnější možné – svět jen zní jinak a bolí mě uši</li>
</ul>
</li>
<li>Když dám na chvíli nabít kdykoliv mě to náhodou napadne, přijde mi, že se nikdy nevybijí</li>
<li>Často umí fungovat vypnuté (vybité) přes 3.5 jack kabel</li>
<li>Moje současné a nejoblíbenější Marshall II Bluetooth
<ul>
<li>Zkoušel jsem i Marshall III Bluetooth, má lepší Bluetooth, ale horší mechanismus sklápění</li>
</ul>
</li>
</ul>
<h2 id="spojená-drátem">Spojená drátem</h2>
<ul>
<li>Když není možnost si drátek za krkem přitáhnout na těsno, při běhu skáče na krku a je to nepříjemné</li>
<li>V uších velmi lehké, je část s baterkou a ovládacími tlačítky dostatečně nízko, aby nevisely ve vzduchu</li>
<li>Potřeba si vypěstovat reflex dat je každý večer nabít</li>
<li>Můžou vypadat jako úplně normální “klasická kabelová” sluchátka</li>
</ul>
<h2 id="špunty-oddělené">Špunty oddělené</h2>
<ul>
<li>Těžší v uších</li>
<li>Mě dost vypadávaly, některé mají “pomocné výplně” do tragusu, pak (asi) drží lépe</li>
<li>Průběžné nabíjení je dost dobré, jen nejde poslouchat mnoho hodin v kuse</li>
<li>Některé typy hrozně trčí z uší – spíš <em>fashion choice</em></li>
</ul>
<h1 id="příklady-použití">Příklady použití</h1>
<h2 id="chci-sluchátka-1">Chci sluchátka #1</h2>
<ul>
<li>Na doma, abych u toho mohl pracovat</li>
<li>Na cesty MHD, vlakem, letadlem
=> Přes hlavu</li>
<li>Spíš menší, skladné, odolné
=> Na uši</li>
<li>Mám Bluetooth 4.2
=> Kupuji 4.2 / 5.0</li>
<li>Rozpočet do 2 000 Kč</li>
</ul>
<figure>
<a href="../../images/2019/09/filtery1.png"><img src="../../images/2019/09/filtery1.png" /></a>
<figcaption>Filtery vytvořené na základě požadavků<br />
Source: <a href="https://bezdratova-sluchatka.heureka.cz">Heureka</a></figcaption>
</figure>
<p>Z obrázku nepoznám</p>
<ul>
<li>Jak dobře mi sednou</li>
<li>Jak bytelný je skládací mechanismus
=> Objednám prvních několik kusů, třeba ob 3 dny, abych mi doma všechny netrčely naráz</li>
</ul>
<figure>
<a href="../../images/2019/09/selectedHeadphones1.png"><img src="../../images/2019/09/selectedHeadphones1.png" /></a>
<figcaption>Které suchátka si minimálně objednám<br />
Source: <a href="https://bezdratova-sluchatka.heureka.cz">Heureka</a></figcaption>
</figure>
<p>Snad to pomohlo a přeji zdárné vybírání na <a href="https://www.heureka.cz/">Heurece</a>.</p>Ronald LucŘada otázek na Heureka filteryBits&Pretzels 20192019-09-26T06:47:00+00:002019-09-26T06:47:00+00:00https://ronaldluc.com/events/bits-and-pretzels<h1 id="opening-ceremony">Opening ceremony</h1>
<h2 id="rossmann--linkedin-co-founder-from-blitzscaling">Rossmann – LinkedIn co-founder from BlitzScaling</h2>
<ul>
<li>BlitzScaling is scaling speed at the cost of efficiency or capital
<ul>
<li>Don’t solve problems with bad management, tooling, tech-dept etc. Focus on scale</li>
<li>The reason is to take over the competition</li>
<li>It is expensive – don’t BlitzScale when there is no competition</li>
</ul>
</li>
</ul>
<h3 id="companystartup-size">Company/startup size</h3>
<ul>
<li>Family – 1–10
<ul>
<li>Searching for market fit</li>
</ul>
</li>
<li>Tribe – 10–100
<ul>
<li>Start scaling, market fit should be clear</li>
</ul>
</li>
<li>Village – 100-1000
<ul>
<li>BlitzScaling at full scale,</li>
</ul>
</li>
<li>Nation – 1000-10000
<ul>
<li>BlitzScaling only with a totally new idea/product</li>
<li>Generally going back efficiency</li>
</ul>
</li>
</ul>
<h2 id="elsa-barnadotte-from-karma">Elsa Barnadotte from Karma</h2>
<ul>
<li>Mission to turn food waste into profit</li>
<li>Let restaurants and shops easily sell soon-to-be-waste for 1/2 the price with <em>Karma</em></li>
<li>5 % of Sweden already uses</li>
</ul>
<p>My view as a Czech:</p>
<ul>
<li>It will fail in many places, one Sushi place in Brno tried to sell for half in the last 30 minutes</li>
<li>Result: people stoped going to the restaurant in the last 3 hours and waited in a line for the 30 minutes of sale :(</li>
</ul>
<h2 id="openbiotics">OpenBiotics</h2>
<ul>
<li>Rethink the already used</li>
</ul>
<h2 id="zarah-bruhn-from-social-bee">Zarah Bruhn from Social Bee</h2>
<ul>
<li>Company employing refugees for 1 year</li>
<li>After that <em>let them fly</em> away to companies on their own</li>
<li>Zarah had the great courage, quit school, took pension fund to help others => over 200 refugees being helped at a time within 2 years after “startup” creation</li>
<li>Business can be combined with good</li>
</ul>
<h2 id="frank-niederlander-from-">Frank Niederlander from ???</h2>
<ul>
<li>Founder != responsible leader</li>
</ul>
<h2 id="barack-obama-from-mr-president">Barack Obama from Mr. President</h2>
<blockquote>
<p>10 minutes of warming up, singing and introductions</p>
</blockquote>
<p>How to contact him?</p>
<ul>
<li>Contacted his agency</li>
<li>Talked to many people from his inner circle</li>
<li>They made custom video just for Barack
<ul>
<li>90 seconds about what Bits&Pretzels is</li>
<li>Offered him a traditional Balvarian LederHose</li>
</ul>
</li>
<li>They did not reply</li>
<li>Tried to put deadlines on it
<ul>
<li>Deadlines passed, so they pushed them</li>
</ul>
</li>
<li>They asked them to stop
<ul>
<li>They persisted!
=> <em>It worked!</em></li>
</ul>
</li>
</ul>
<h3 id="the-barák-himself">The Barák himself</h3>
<ul>
<li>How did you help diversity in your presidency?
<ul>
<li>The greater mix and crosssection of people I had, the more likely it was that we would have fresh approaches on the problem.</li>
<li>You had to be intentional about it [increasing diversity] – these in charge <em>has to reach out</em> and they will find the great potential between the “outcast” groups</li>
<li>If you don’t reach out, the discrimination will come automatically – for example just because the of the culture view</li>
<li>How to make sure that the outsiders are not only <em>in the room</em>, but also <em>speaking</em></li>
</ul>
</li>
</ul>
<blockquote>
<p>Use your ears not only your mouth and you might learn something</p>
</blockquote>
<ul>
<li>One of the current problems is climate change – there was big strike just last week. What do you think could enteprenours to do to help?
<ul>
<li>You look like an educated audience, so I think I don’t have to convince you that it is real.</li>
<li>Brita asked before the UN meeting in New York:
<ul>
<li>“Do you think the other world leaders understant the science of Climate change?”</li>
</ul>
</li>
<li>I replied: I am sure they do not understand it at a level that you do.</li>
<li>It is just not enough to start working on Climate Change tech, but you need to talk about it and support in on the Govenmertal level.
<ul>
<li>The gas and oil received governmential support until they were fully integrated</li>
</ul>
</li>
</ul>
</li>
<li>Back to your meeting with Greta. It blew up on Instagram – what kind of leader is she? I mean, she is only 16.
<ul>
<li>She’s extremely young. <em>Kids should not have to do this.</em> There are adults. We have responsibilities that we don’t meet.</li>
<li>We shouldn’t make it that hard for young leaders to have impact.</li>
<li>We did a meeting with the young leaders in Berlin and I saw that most of the people working on climate change here [Europe] are closely connected to the ones doing the same in the US</li>
</ul>
</li>
</ul>
<blockquote>
<p>Kids should not have to do this. [Solve the climate change]</p>
</blockquote>
<ul>
<li>How can you create enteprenourship anywhere?
<ul>
<li>I don’t think we can replicate the type of ecosystem that was created by itself in the Silicon Valley</li>
<li>I believe we can partially recreate and brew entepretional spirit in the University environment</li>
<li>If the spending is not provided to the academies, it cannot be transmited to the technology and innovation</li>
<li>States should be strategic about places that have great startup potential, but cannot raise the funds => wasted potential</li>
<li>We like to romanticize the idea of Anarchy and freedom, but the countries with no state are failed states with no roads, no police and no education</li>
<li>It is hard for small companies to break through before they get crushed
<ul>
<li>Finding the balance in state involvement is a hard problem</li>
</ul>
</li>
</ul>
</li>
<li>Do you think the big companies should be broken up then?
<ul>
<li>I think there should be conversation – ideally started by the companies themselves</li>
<li>As we approach the edge of AI, there should be a lot of thought put into it
<ul>
<li>We want the benefits of those technologies</li>
<li>=> We need a conversation: How to let those companies make the profit, yet benefit all</li>
</ul>
</li>
</ul>
</li>
<li>Do you have and example?
<ul>
<li>I know the founders of AirBnB very well
<ul>
<li>They are not public yet => how do they change their vision for people not to share their houses, but also the experiences?</li>
<li>How to make sure not to lose those visions when you become multi billion?
<ul>
<li>Not loosing the vision is really important for hiring talent</li>
</ul>
</li>
</ul>
</li>
<li>There are big questions how to take the route the</li>
</ul>
</li>
</ul>
<blockquote>
<p>Not loosing the vision is really important for hiring talent</p>
</blockquote>
<ul>
<li>We talked about the bad part of technology, but it can also improve the lives. For example AI.
<ul>
<li>With AI, for example we have ML systems to achieve way better efficiencies for heating and cooling than any engineers could</li>
<li>Or Crisper, to move away from AI
<ul>
<li>It is now so cheap and easy that it is easily in wrong hands</li>
</ul>
</li>
</ul>
</li>
<li>New behavior that improved your live
<ul>
<li>Sleeping – it’s like a drug! I feel happier</li>
<li>More time to read without being focused on a specific subject
<ul>
<li>As a president, there is always a problem in you inbox</li>
<li>The day-to-day consumes your live</li>
</ul>
</li>
</ul>
</li>
<li>I hang out with my wife more
<ul>
<li>It was hard to catch with her since she was still angry of me running for the president</li>
</ul>
</li>
<li>You also wrote a book, right?
<ul>
<li>That has not imporved my life!</li>
<li>The writing is misserable, I want to have it written</li>
</ul>
</li>
<li>We live in a time where there is a lot of nationalism and populism on both sides of Atlantic. How do we deal with it and join/pull people together?
<ul>
<li>Some people think:
<ul>
<li>My status comes from how many people are under me, how many do I dominate</li>
</ul>
</li>
<li>At Enlightment a new idea emerged:
<ul>
<li>We should respect each other even though we are different</li>
<li>Especially in Europe, this approach brought people to higher prosperity</li>
</ul>
</li>
<li>We have to recognize that some mistakes we re made after WW1, WW2, when joining Germany together</li>
<li>The globalisation makes more winners but also more losers
<ul>
<li>More thinking should have been put into it</li>
</ul>
</li>
<li>Peoples traditions, culture, gets threatened and we have to respect that</li>
<li>I think the earlier thinking leads only to more war and conflict
<ul>
<li>Europe knows the best where that leads</li>
</ul>
</li>
<li>There has to be more weight put on the facts</li>
<li>The thing is:
<ul>
<li>If you are the best at something, this is a great time for you. You can go global.</li>
<li>Not all have the skills and means. There will be backlash for all of the people put behind.</li>
</ul>
</li>
</ul>
</li>
<li>
<QUESTION>
</QUESTION>
<ul>
<li>If there is change in society, we have to change our institutions
<ul>
<li>Change the taxes, change the politics, change the focus</li>
<li>There never was a society where everyone could do whatever</li>
</ul>
</li>
</ul>
</li>
</ul>
<blockquote>
<p>I am optimistic. If your name is Barack Husain Obama and you are in the White House, you must be optimistic</p>
</blockquote>
<ul>
<li>And why are you so optimistic?
<ul>
<li>I am optimistic because of the young people
<ul>
<li>Whereever I go, there are young people who go with courage and innovation and solve big problems –</li>
</ul>
</li>
<li>If you had to choose a time to be born at, but you did not know where and as who you would be born, you would, on average, choose today.</li>
</ul>
</li>
</ul>
<h1 id="evening">Evening</h1>
<h2 id="bavarian-champions">Bavarian Champions</h2>
<ul>
<li>Daniel Kraus from Founder of FLixBus</li>
<li>Andreas Bodczek from CEO IDnow</li>
<li>Markus Dickhardt from Co-Founder Roadsurfer from renting vans</li>
<li>
<p>Andreas Bruckschogl from Co-Founder Ryte from automated website quality assurance</p>
</li>
<li>Why is Munich a good place to start a startup?
<ul>
<li>5 times more funding than in Berlin</li>
<li>More senior people, because of corporates</li>
<li>Mountains and other nature</li>
<li>Young talent driven for the mountains, skying etc.</li>
</ul>
</li>
<li>Why not?
<ul>
<li>Hard to get started for foreighners
<ul>
<li>Hard to start a company</li>
<li>Extra hard to get a flat</li>
</ul>
</li>
</ul>
</li>
<li>Do you plan to expand abroad?
<ul>
<li>[FlixBus] US and Asia are our next targets for the busses</li>
<li>[FlixBus] Trains are the main and big focus for Germany and Europe</li>
</ul>
</li>
<li>Is there a wording for Munich that you would love to see? Something like Silicon Valey, Silicon Beach
<ul>
<li>Silicon Avalanche? :D</li>
<li>[FlixBus] I don’t believe it’s needed. You need to do serious work, actual work. With a name comes hype</li>
</ul>
</li>
</ul>
<h2 id="founder-of-dropbox">Founder of DropBox</h2>
<ul>
<li>How did you start?
<ul>
<li>DropBox was founded 14 years ago, but I got my first programming job when I was 14 because I found a Game bug</li>
<li>Noone is born as a CEO</li>
<li>Every 14 months your job as CEO changes completely – as if you were fired and rehired to a different company</li>
<li>The good thing is, none of those changes happen over night, but day after day</li>
<li>The thing that keeps me up is reading
<ul>
<li>Everyone thinks his situation is unique, but it is not. There are books about it.</li>
</ul>
</li>
</ul>
</li>
</ul>
<blockquote>
<p>Noone is born as a CEO</p>
</blockquote>
<ul>
<li>Is there something you miss from starting the company («<)
<ul>
<li>I am actually coding again after many years – on my free time for fun
<ul>
<li>There is whole renesaince («<) in ML</li>
</ul>
</li>
<li>I stopped coding few years ago, since at some point you need to give it to SW developer team</li>
<li>Now there is fresh air, because we just introduced a new product <INSERT>
</INSERT> <ul>
<li>It is focused on calmness and everything being well organized</li>
</ul>
</li>
<li>I met a <POSITION> from SpaceX and asked him: Which tooling do you use? What does it take to put someone on the Mars?
</POSITION> <ul>
<li>A lot of emails and a lot of files</li>
</ul>
</li>
<li>And then I realised that we need to solve it. Emails did not change since we had 5 of them per week.</li>
<li>We are the most productive if we are not interrupted 200 times a day</li>
<li>No meeting Wednesday – an idea going around Silicon Valley for quite some time
<ul>
<li>It’s great! So much done.</li>
</ul>
</li>
</ul>
</li>
</ul>
<blockquote>
<p>As a CEO, you have 100× more tasks than time. Yet all of the great enteprenours had the same 24 hours a day you have.</p>
</blockquote>
<ul>
<li>I remember when I visited my dad at work 20 years ago
<ul>
<li>Many things were the same, yet it was so different.</li>
<li>He put his workcase at the door and did not think of</li>
</ul>
</li>
</ul>
<p>.</p>Ronald LucNotes from talks and Mr. PrezidentGeneric Neural Elastic Search2019-09-20T06:47:00+00:002019-09-20T06:47:00+00:00https://ronaldluc.com/events/chinese-ai-landscape<h3 id="han-xiao"><a href="https://www.linkedin.com/in/hxiao87/">Han Xiao</a></h3>
<ul>
<li>Working at Tencent AI</li>
<li>Successful OpenSource projects he worked on
<ul>
<li><a href="https://github.com/zalandoresearch/fashion-mnist">Fashion-MNIST</a></li>
<li><a href="https://github.com/hanxiao/bert-as-service">Bert-as-service</a></li>
</ul>
</li>
</ul>
<h3 id="tencent--10">Tencent != 10¢</h3>
<ul>
<li>From Internet to Wechat</li>
<li>Basically Google for China + they managed to be good at social media and messaging</li>
<li>They have a lot of data and they can probably use it, because there is no GDPR in China</li>
</ul>
<h3 id="tencent-ai">Tencent AI</h3>
<ul>
<li>70 AI scientists + 300 app developers (== ML Engineers)</li>
<li>The #1 in AI publications in China, yet still 1/10 of Google papers</li>
</ul>
<figure>
<a href="../../images/2019/09/gnes-banner.png"><img src="../../images/2019/09/gnes-banner.png" /></a>
<figcaption>Source: <a href="https://github.com/gnes-ai/gnes">github/GNES</a></figcaption>
</figure>
<h1 id="gnes--generic-neural-elastic-search">GNES ‒ Generic Neural Elastic Search</h1>
<ul>
<li>
<h2 id="one-framework-to-train-text-image-and-video-search">One framework to train <em>text, image</em> and <em>video</em> search</h2>
</li>
<li>Should be used as TF, just everything deployed in a different place</li>
<li>A bit like KuberFlow/AirFlow, but optimized for Search scenario</li>
</ul>
<h3 id="premises">Premises</h3>
<ul>
<li>Cloud native</li>
<li>Semantic search (using DNN)</li>
<li>End2end</li>
</ul>
<h3 id="obstacles">Obstacles</h3>
<ul>
<li>What is the distance metric for doc vectors?</li>
<li>How to handle the difference between short / long texts (video, images)</li>
</ul>
<h3 id="the-idea">The Idea</h3>
<ul>
<li>Minimum information unit × minimum semantic unit × optimum semantic unit</li>
<li>a word × a sentence × ???</li>
<li>a pixel × a 64×64 patch × ???</li>
<li>a pixel in 1 frame × a 2-3 sec shot × ???</li>
<li>What are the optima?
<ul>
<li>Do an experiment!
<ul>
<li>=> determined by ONE preprocessor</li>
</ul>
</li>
</ul>
</li>
<li>Models change too quickly
<ul>
<li>=> plug them in as a Docker container</li>
</ul>
</li>
</ul>
<h3 id="specs">Specs</h3>
<ul>
<li>Everything is its own microservice</li>
<li>All of the components are defined by a YAML file => immutable code, just change the YAML</li>
</ul>
<figure>
<a href="../../images/2019/09/gnes-component-overview.png"><img src="../../images/2019/09/gnes-component-overview.png" /></a>
<figcaption>Overview of possible parts<br />Source: <a href="https://github.com/gnes-ai/gnes">github/GNES</a></figcaption>
</figure>
<h1 id="questions">Questions</h1>
<ul>
<li>Are you happy with the “All as microservice” design?</li>
<li>Do you have any idea, what is the overhead?
<ul>
<li>It adds overhead if the processing of each component is small
<ul>
<li>=> great for images or better video (just FFMPeg)</li>
</ul>
</li>
</ul>
</li>
<li>Is the presentation somewhere?
<ul>
<li>Maybe</li>
</ul>
</li>
<li>Can it be put into lambda functions?
<ul>
<li>Not for now</li>
<li>Not because of the loading</li>
</ul>
</li>
</ul>
<hr />
<ul>
<li>How to address multiple languages – when preprocessing is completely different?
<ul>
<li>It is your job to write a specific YAML file that preprocesses it</li>
</ul>
</li>
<li>How to handle huge models in Docker images
<ul>
<li>Just put everything into the container</li>
</ul>
</li>
<li>What is the view in China on big OpenSource contributions?
<ul>
<li>They do not get much support for OpenSource (a lot of papering, what to publish etc.)</li>
<li>30 % work the relationship making</li>
</ul>
</li>
<li>Challenges for Europe with regards to AI
<ul>
<li>A lot of Chinese AI is focused on the consumer (you can test something on this milion of data and that milion of data)</li>
<li>=> not much 0 to 1 research, a lot of 1 to N research</li>
<li>Don’t try to copy B2C of US and China</li>
<li>Try to focus on the middle man, companies, which can change the world a lot, but are not “shiny” (no coverage)
<ul>
<li>Upgrade the traditional industry! Keep Europe great still!</li>
</ul>
</li>
</ul>
</li>
</ul>Ronald Lucand Chinese AI landscape -- notes from a talk by Han XiaoLetná, kdo tam byl?2019-06-24T06:47:00+00:002019-06-24T06:47:00+00:00https://ronaldluc.com/democracy/letna-protests<p>Velmi přibližný odhad průměrného věku demonstrantů na Letné:</p>
<figure>
<a href="../../images/2019/06/letna_faces.png"><img src="../../images/2019/06/letna_faces.png" /></a>
<figcaption>Zdroj: vlastní tvorba</figcaption>
</figure>
<p>Na fotce je 138 obličejů rozpoznaných pomocí <a href="http://max-facial-age-estimator-mini-web-app.max.us-south.containers.appdomain.cloud/">IBM Face Age Estimatoru</a>, průměrný věk je 38.7 let, čili nejproduktivnější část českého obyvateltva a ne “flákající se študáci”.</p>
<p>Kdo nevěří odhadům umělé inteligence, může se na protestanty podívat sám:</p>
<figure>
<a href="../../images/2019/06/letna_ppl.jpg"><img src="../../images/2019/06/letna_ppl.jpg" /></a>
<figcaption>Zdroj: vlastní tvorba</figcaption>
</figure>
<p>A jaká byla atmosféra? Matfyzácká skupinka si vzala štafle, takže jsme mohli nakouknout nad dav.</p>
<!-- Courtesy of embedresponsively.com //-->
<div class="responsive-video-container">
<iframe src="https://www.youtube.com/embed/q125QdG4lN4" frameborder="0" allowfullscreen=""></iframe>
</div>
<p>Za co především jsem byl na Letné já osobně:</p>
<ul>
<li>Dosazení Marie Benešové, když Jan Kněžínek odmítl <em>poslouchat</em> (<a href="https://www.respekt.cz/politika/kauza-benesova-je-nejzasadnejsi-zalezitosti-od-babisova-vstupu-do-politiky">Respekt</a>, <a href="https://ct24.ceskatelevize.cz/domaci/2801414-marie-benesova-se-vraci-jako-ministryne-spravedlnosti-jeji-jmeno-budi-kritiku">ČT</a>, <a href="https://www.idnes.cz/zpravy/domaci/knezinek-ministr-spravedlnosti-andrej-babic-vymena-ministra.A190419_083544_domaci_linv">iDNES</a>)</li>
<li>450 milionů Kč o které Agrofert <em>okradl</em> malé a střední podniky (<a href="https://zpravy.aktualne.cz/domaci/zprava-olaf-capi-hnizdo-kompletni-andrej-babis/r~e4e4d5c8f62111e7984d0cc47ab5f122/">přeložené kompletní znění</a>, <a href="https://www.respekt.cz/politika/babis-a-dotace-velke-shrnuti-v-otazkach-a-odpovedich">shrnutí Respekt</a>, <a href="https://www.irozhlas.cz/zpravy-domov/andrej-babis-stret-zajmu-audit-zprava-politici-lidovci-pirati_1906011423_lac">iROZHLAS</a>)</li>
<li>Vyšetřované poslání vlastního syna na smrt na občanskou válkou zmítaný Krym (<a href="https://www.youtube.com/watch?v=Hn52PRGRIDM">DVTV</a>, <a href="https://www.denik.cz/z_domova/babisuv-syn-mel-podstoupit-nedobrovolny-zakrok-napsal-to-v-e-mailu-policii-20181129.html">denik.cz</a>, <a href="https://zpravy.aktualne.cz/domaci/babisuv-syn-tvrdi-ze-ho-lide-napojeni-na-kauzu-capi-hnido-od/r~1573860ee6b711e8b9390cc47ab5f122/">aktuálně.cz</a>)</li>
<li>Střet zájmů tak jasný, že si Andrej dovolí zajet za vrchním komisařem <em>“to vyřídit”</em> (<a href="https://www.idnes.cz/zpravy/domaci/andrej-babis-zprava-ke-stretu-zajmu-evropska-komise-summit-eu.A190605_082626_domaci_niv">iDNES</a>, <a href="https://www.novinky.cz/domaci/506366-agrofert-musi-vratit-vsechny-evropske-dotace-od-unora-2017-babis-je-ve-stretu-zajmu.html">novinky.cz</a>)</li>
</ul>
<h3 id="ai-simulace-nadcházející-listopadové-demonstrace">AI simulace nadcházející <em>listopadové</em> demonstrace</h3>
<!-- Courtesy of embedresponsively.com //-->
<div class="responsive-video-container">
<iframe src="https://www.youtube.com/embed/yxmthRAIrGM" frameborder="0" allowfullscreen=""></iframe>
</div>
<p>Pokud stojíte o Česko s nezávislou justicí, přidejte se za pár měsícu ke statisícům <em>nemlčících ovcí</em> znovu na Letné.</p>Ronald LucA proč jsem tam byl já?Beyond MobileNet2019-03-04T06:47:00+00:002019-03-04T06:47:00+00:00https://ronaldluc.com/ai/beyond-mobilenet<p>MobileNet v2<sup id="fnref:f" role="doc-noteref"><a href="#fn:f" class="footnote" rel="footnote">1</a></sup> is state-of-the-art in precision for models targeted for real-time processing. It doesn’t reach the FPS of Yolo v2/v3 (Yolo is 2–4 times faster, depending on implementation). That said let’s think about some upgrades that would make a MobileNet v3.</p>
<h2 id="upgrade-the-dataset">Upgrade the dataset</h2>
<p>DNNs are often held back by the dataset, not by the model itself. The creation (and annotation) of real datasets with high diversity is hard and costly. The proposed idea from NVIDIA is to pretrain the model with synthetic data<sup id="fnref:fa" role="doc-noteref"><a href="#fn:fa" class="footnote" rel="footnote">2</a></sup>.</p>
<p>This has been done before, but NVIDIA examines the effects of randomization beyond the test set domain. The added diversity in pretrain phase increases the performance by a significant margin especially for the SSD.</p>
<p>Results for KITTI validation set pretrained on Virtual KITTI or NVIDIA synthetic data:</p>
<table>
<thead>
<tr>
<th style="text-align: center">Architecture</th>
<th style="text-align: center">Virtual KITTI</th>
<th style="text-align: center">NVIDIA</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">Faster R-CNN</td>
<td style="text-align: center"><strong>79.7</strong></td>
<td style="text-align: center">78.1</td>
</tr>
<tr>
<td style="text-align: center">R-FCN</td>
<td style="text-align: center">64.6</td>
<td style="text-align: center"><strong>71.5</strong></td>
</tr>
<tr>
<td style="text-align: center">SSD</td>
<td style="text-align: center">36.1</td>
<td style="text-align: center"><strong>46.3</strong></td>
</tr>
</tbody>
</table>
<p>As shown in the chart below, unrealistic textures and different lighting conditions have the most significant impact.</p>
<figure>
<a href="../../images/2019/03/randomization-effects.png"><img src="../../images/2019/03/randomization-effects.png" /></a>
<figcaption>Impact upon AP by omitting individual randomized components from the DR data generation procedure. Source: NVIDIA </figcaption>
</figure>
<p>Examples of synthetic scenes:</p>
<figure>
<a href="../../images/2019/03/synthetic-scene-1.png"><img src="../../images/2019/03/synthetic-scene-1.png" /></a>
<figcaption>Source: NVIDIA </figcaption>
</figure>
<figure>
<a href="../../images/2019/03/synthetic-scene-2.png"><img src="../../images/2019/03/synthetic-scene-2.png" /></a>
<figcaption>Source: NVIDIA </figcaption>
</figure>
<p>On the cars in KITTI dataset the Avg. Precision @ .50 IoU increases from 96.5 to 98.5 in other words <strong>the average error is reduced by 57 %</strong> by the usage of synthetic data + 6000 real images.</p>
<h2 id="faster-multi-scaling">Faster multi-scaling</h2>
<p>In the original paper introducing MobileNetV2<sup id="fnref:fb" role="doc-noteref"><a href="#fn:fb" class="footnote" rel="footnote">3</a></sup> <strong>including multi-scale inputs and adding leftright flipped images</strong> (MF) and decreasing the output stride (OS) increases the mean IoU from 75.32 % to 77.33 %. Adding Atrous Spatial Pyramid Pooling (ASPP) increases it further to 78.42 %. The researchers argue not to include due to the almost 100× increase in the multiply-accumulate operations (MAdds).</p>
<p>Results for PASCAL VOC 2012 validation set:</p>
<table>
<thead>
<tr>
<th style="text-align: center">OS</th>
<th style="text-align: center">ASPP</th>
<th style="text-align: center">MF</th>
<th style="text-align: center">mIoU</th>
<th style="text-align: center">MAdds</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">16</td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center">75.32</td>
<td style="text-align: center"><strong>2.75B</strong></td>
</tr>
<tr>
<td style="text-align: center">8</td>
<td style="text-align: center"> </td>
<td style="text-align: center">✓</td>
<td style="text-align: center">77.33</td>
<td style="text-align: center">152.6B</td>
</tr>
<tr>
<td style="text-align: center">8</td>
<td style="text-align: center">✓</td>
<td style="text-align: center">✓</td>
<td style="text-align: center"><strong>78.42</strong></td>
<td style="text-align: center">387B</td>
</tr>
</tbody>
</table>
<p>To harnest the advantages of multi-scale processing while keeping the MAdds low I propose to incorporate the recently published TridentNet architecture<sup id="fnref:fc" role="doc-noteref"><a href="#fn:fc" class="footnote" rel="footnote">4</a></sup>. It works by proposing independent predictions for different sizes of objects.</p>
<figure>
<a href="../../images/2019/03/different-types-of-multi-size.png"><img src="../../images/2019/03/different-types-of-multi-size.png" /></a>
<figcaption>Source: TridentNet</figcaption>
</figure>
<p>Results for COCO test-dev set:</p>
<table>
<thead>
<tr>
<th style="text-align: right">Method</th>
<th style="text-align: right">Backbone</th>
<th style="text-align: right">AP</th>
<th style="text-align: right">AP @ 0.5</th>
<th style="text-align: right">AP small</th>
<th style="text-align: right">AP medium</th>
<th style="text-align: right">AP large</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">SSD513</td>
<td style="text-align: right">ResNet-101</td>
<td style="text-align: right">31.2</td>
<td style="text-align: right">50.4</td>
<td style="text-align: right">10.2</td>
<td style="text-align: right">34.5</td>
<td style="text-align: right">49.8</td>
</tr>
<tr>
<td style="text-align: right">Deformable R-FCN</td>
<td style="text-align: right">Aligned-Inception-ResNet-Deformable</td>
<td style="text-align: right">37.5</td>
<td style="text-align: right">58.0</td>
<td style="text-align: right">19.4</td>
<td style="text-align: right">40.1</td>
<td style="text-align: right">52.5</td>
</tr>
<tr>
<td style="text-align: right">SNIPER</td>
<td style="text-align: right">ResNet-101-Deformable</td>
<td style="text-align: right">46.1</td>
<td style="text-align: right">67.0</td>
<td style="text-align: right">29.6</td>
<td style="text-align: right">48.9</td>
<td style="text-align: right">58.1</td>
</tr>
</tbody>
<tbody>
<tr>
<td style="text-align: right">TridentNet</td>
<td style="text-align: right">ResNet-101</td>
<td style="text-align: right">42.7</td>
<td style="text-align: right">63.6</td>
<td style="text-align: right">23.9</td>
<td style="text-align: right">46.6</td>
<td style="text-align: right">56.6</td>
</tr>
<tr>
<td style="text-align: right">TridentNet*</td>
<td style="text-align: right">ResNet-101-Deformable</td>
<td style="text-align: right"><strong>48.4</strong></td>
<td style="text-align: right"><strong>69.7</strong></td>
<td style="text-align: right"><strong>31.8</strong></td>
<td style="text-align: right"><strong>51.3</strong></td>
<td style="text-align: right"><strong>60.3</strong></td>
</tr>
</tbody>
</table>
<h2 id="time-coherent-predictions">Time coherent predictions</h2>
<p>The first shot on minimizing the frame-to-frame inconsistency – the idea is inserting the information from last frame into the new frame processing. If <strong>the insertion is done at the linear bottlenecked layer</strong>, the added processing complexity for merging is minor in comparison to the convolutional layers.</p>
<figure>
<a href="../../images/2019/03/time-coherent-mobilenet-1.png"><img src="../../images/2019/03/time-coherent-mobilenet-1.png" /></a>
<figcaption>Source: own work</figcaption>
</figure>
<figure>
<a href="../../images/2019/03/time-coherent-mobilenet-2.png"><img src="../../images/2019/03/time-coherent-mobilenet-2.png" /></a>
<figcaption>Source: own work</figcaption>
</figure>
<h2 id="conclusions">Conclusions</h2>
<p>By implementing 3 higher outlined improvements to the dataset and the architecture, there is a theoretical potential for improvement of AP between 4 and 15 % depending on the task. Introduction of time coherence can reduce the prediction flickering and thus increase the ease of further use.</p>
<h2 id="references">References</h2>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:f" role="doc-endnote">
<p>Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. <a href="https://arxiv.org/abs/1801.04381">arXiv:1801.04381</a> <a href="#fnref:f" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fa" role="doc-endnote">
<p>Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, Stan Birchfield. 2018. Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization. <a href="https://arxiv.org/abs/1804.06516.pdf">arXiv:1804.0651</a> <a href="#fnref:fa" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fb" role="doc-endnote">
<p>Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. <a href="https://arxiv.org/abs/1801.04381">arXiv:1801.04381</a> <a href="#fnref:fb" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fc" role="doc-endnote">
<p>Yanghao Li, Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang. 2019. Scale-Aware Trident Networks for Object Detection. <a href="https://arxiv.org/abs/1901.01892">arXiv:1901.01892</a> <a href="#fnref:fc" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Ronald LucIdeas for improvementA cheap security camera footage classification2019-03-04T06:47:00+00:002019-03-04T06:47:00+00:00https://ronaldluc.com/ai/small-image-classification<p>Imagine we have a huge <em>Input dataset</em> of small images from a <strong>cheap security cameras</strong>. There are several obstacles with images in this dataset:</p>
<ul>
<li>Small size (under 200 px × 200 px)</li>
<li>Grayscale from both visible and infrared spectrum, unsorted</li>
<li>Partial labels with “meta categories” (human, animal, wind, nothing)</li>
</ul>
<h2 id="datasets">Datasets</h2>
<p>To overcome those obstacles, I propose to use the <em>Input dataset</em> only as a validation set (or a part of it to re-train the model). The difference between a deer, a rabbit, a sheep and a snake in comparison to the category “human” is significant enough not to be classified only as an “animal”.</p>
<p>Very large high-res labeled RGB datasets of all needed animals are easy to find: for example Animals with attributes 2<sup id="fnref:fa" role="doc-noteref"><a href="#fn:fa" class="footnote" rel="footnote">1</a></sup> (37322 images of 50 animals), parts of other datasets or a new one using a web scrape.</p>
<figure>
<a href="../../images/2019/03/overview-of-animals-with-attributes-2.png"><img src="../../images/2019/03/overview-of-animals-with-attributes-2.png" /></a>
<figcaption>Source: <a href="https://arxiv.org/abs/1707.00600">Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly</a></figcaption>
</figure>
<p>More challenging is getting a large, high-res labeled IR image dataset. There are plenty of IR datasets, but not in the same domain and for the same purpose as we want. My solution is to use the (RGB, IR) and (RGB, Near IR) pairs datasets to train a conditional GAN (<a href="/ai/Pix2pix">basic idea behind cGANS</a>).</p>
<figure>
<a href="../../images/2019/03/ranus-rgb-and-nir-urban-scene-dataset-for-deep-scene-parsing.png"><img src="../../images/2019/03/ranus-rgb-and-nir-urban-scene-dataset-for-deep-scene-parsing.png" /></a>
<figcaption>Source: <a href="https://joonyoung-cv.github.io/assets/paper/18_ral_ranus.pdf">RANUS: RGB and NIR Urban Scene Dataset for Deep Scene Parsing</a></figcaption>
</figure>
<figure>
<a href="../../images/2019/03/rgb-infrared-cross-modality-person-re-identification.png"><img src="../../images/2019/03/rgb-infrared-cross-modality-person-re-identification.png" /></a>
<figcaption>Source: <a href="https://www.eecs.qmul.ac.uk/~sgg/papers/WuEtAl_ICCV2017.pdf">RGB-Infrared Cross-Modality Person Re-Identification</a></figcaption>
</figure>
<figure>
<a href="../../images/2019/03/rgb-nir-scene-dataset-image-and-visual-representation-lab-ivrl.png"><img src="../../images/2019/03/rgb-nir-scene-dataset-image-and-visual-representation-lab-ivrl.png" /></a>
<figcaption>Source: <a href="https://ivrl.epfl.ch/research-2/research-downloads/supplementary_material-cvpr11-index-html/">RGB-NIR Scene Dataset from IVRL, EPFL</a></figcaption>
</figure>
<p>Many more (RGB, IR) and (RGB, NIR) can be found in <a href="http://homepages.inf.ed.ac.uk/rbf/CVonline/Imagedbase.htm">CVonline: Image Databases</a>.
To bring this afford to another level, the best would be to create own (RGB, IR) camera rig and <strong>add a small but very similar in domain dataset.</strong> This could be done in one day at a trip to a zoo, forests and plains around the city.</p>
<figure>
<a href="../../images/2019/03/rgb-nir-camera-rig.png"><img src="../../images/2019/03/rgb-nir-camera-rig.png" /></a>
<figcaption>Example of a sufficient camera rig. Source: <a href="https://joonyoung-cv.github.io/assets/paper/18_ral_ranus.pdf">RANUS: RGB and NIR Urban Scene Dataset for Deep Scene Parsing</a></figcaption>
</figure>
<h3 id="datasets-overview">Datasets overview</h3>
<figure>
<a href="../../images/2019/03/datasets-overview.png"><img src="../../images/2019/03/datasets-overview.png" /></a>
<figcaption>Source: own work</figcaption>
</figure>
<h2 id="processing">Processing</h2>
<p>In proposed models, I will be using following computation blocks:</p>
<ul>
<li><strong>Res Reduct</strong> – for downsampling</li>
<li><strong>Color Reduction</strong> – for color dim reduction to match <em>Input data</em></li>
<li><strong>rgb2ir cGAN</strong> – basic Pix2pix<sup id="fnref:fb" role="doc-noteref"><a href="#fn:fb" class="footnote" rel="footnote">2</a></sup>, the same using BigGAN<sup id="fnref:fc" role="doc-noteref"><a href="#fn:fc" class="footnote" rel="footnote">3</a></sup>, or incorporate style AdaIN from StyleGAN<sup id="fnref:fd" role="doc-noteref"><a href="#fn:fd" class="footnote" rel="footnote">4</a></sup></li>
<li><strong>Super Resolution</strong> – LapSRN<sup id="fnref:fe" role="doc-noteref"><a href="#fn:fe" class="footnote" rel="footnote">5</a></sup>, NVIDIA GAN super res (<a href="https://developer.download.nvidia.com/assets/gameworks/downloads/regular/GDC17/DeepLearning_MaterialsTextures_GDC17_FINAL.pdf?9aNNcCkiIUA-Ae3076830IlFjMoLIUolJkTlXe72yfbccwYfdahRHmg8jXDNrg3y1kbNQ4LeSrMVmlAhUhfLtTkJwB3MzMsA7_vyvi3MfKoCYgc8PE8i8BWvE2oVOHM9lbAz4ikcBM8A35ebapjDJkm2ESDlAPJUJmETLzMZCHqxFhiozGnZpLFmmd-3xgrNeIaqJ2gI">only a presentation</a>)</li>
<li><strong>Conv Classifier</strong> – ResNet, <a href="/ai/beyond-mobilenet">MobileNet v2</a> depending on the inference speed needed</li>
</ul>
<h3 id="super-resolution">Super resolution</h3>
<figure>
<a href="../../images/2019/03/fast-and-accurate-image-super-resolution-with-deep-laplacian-pyramid-networks-2.png"><img src="../../images/2019/03/fast-and-accurate-image-super-resolution-with-deep-laplacian-pyramid-networks-2.png" /></a>
<figcaption>4× super res. Source: <a href="https://arxiv.org/abs/1710.01992">Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks</a></figcaption>
</figure>
<figure>
<a href="../../images/2019/03/fast-and-accurate-image-super-resolution-with-deep-laplacian-pyramid-networks-1.png"><img src="../../images/2019/03/fast-and-accurate-image-super-resolution-with-deep-laplacian-pyramid-networks-1.png" /></a>
<figcaption>Source: <a href="https://arxiv.org/abs/1710.01992">Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks</a></figcaption>
</figure>
<h3 id="processing-overview">Processing overview</h3>
<figure>
<a href="../../images/2019/03/processes-overview.png"><img src="../../images/2019/03/processes-overview.png" /></a>
<figcaption>Source: own work</figcaption>
</figure>
<h2 id="proposed-pipelines">Proposed pipelines</h2>
<p>The first pipeline is divided into 3 segments. The first is a collection of (RGB, IR) or (RGB, NIR) from <strong>different domains</strong> used to train second segment. The Labeled (RGB, IR) pair generator is trained on the out-of-domain pairs to generate a <strong>large amount of domain-specific IR images with labels</strong> (different animals, people, nature). The combined well labeled RGB and IR (or NIR) dataset is used in the classifier to enhance it’s performance.</p>
<figure>
<a href="../../images/2019/03/no-upsample-cgan.png"><img src="../../images/2019/03/no-upsample-cgan.png" /></a>
<figcaption>Source: own work</figcaption>
</figure>
<p>To improve on models performance I propose to use the high resolution of the well labeled domain-specific images. This would be done by training a super resolution network used to upsample the camera images before classification at inference time. Changes to the first pipeline are colored yellow.</p>
<figure>
<a href="../../images/2019/03/upsample-cgan.png"><img src="../../images/2019/03/upsample-cgan.png" /></a>
<figcaption>Source: own work</figcaption>
</figure>
<p>Enhancer could do more than super resolution. Even better, in the time of teaching of the Classifier, merge it with the Enhancer and train both jointly.</p>
<h3 id="sketches-of-another-approaches">Sketches of another approaches</h3>
<ul>
<li>Build a grayscale RGB/IR classifier and then classify them separately without additional IR images generation</li>
<li>Skip the <strong>Unlabeled (RGB, IR) pairs</strong> and the <strong>Labeled (RGB, IR) pairs generator</strong> segment
<ul>
<li>Build the <strong>Enhancer</strong> for super resolution</li>
<li>Train the <strong>Classifier</strong> on only grayscale visible light images with 2–4× res of the <em>Input dataset</em></li>
<li>Re-train the <strong>Classifier</strong> on upscaled <em>Input dataset</em> images</li>
</ul>
</li>
</ul>
<h2 id="conclusions">Conclusions</h2>
<p>…has yet to be implemented!</p>
<h2 id="references">References</h2>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:fa" role="doc-endnote">
<p>Y. Xian, C. H. Lampert, B. Schiele, Z. Akata. “Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly”, IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 40(8), 2018. <a href="https://arxiv.org/abs/1707.00600">arXiv:1707.00600</a> <a href="#fnref:fa" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fb" role="doc-endnote">
<p>Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros. Image-to-Image Translation with Conditional Adversarial Networks. 2016. <a href="https://arxiv.org/abs/1611.07004">arXiv:1611.07004</a> <a href="#fnref:fb" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fc" role="doc-endnote">
<p>Andrew Brock, Jeff Donahue, Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. 2018. <a href="https://arxiv.org/abs/1809.11096">arXiv:1809.11096</a> <a href="#fnref:fc" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fd" role="doc-endnote">
<p>Tero Karras, Samuli Laine, Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. 2018. <a href="https://arxiv.org/abs/1812.04948">arXiv:1812.04948</a> <a href="#fnref:fd" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fe" role="doc-endnote">
<p>Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang. Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks. 2017. <a href="https://arxiv.org/abs/1710.01992">arXiv:1710.01992</a> <a href="#fnref:fe" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Ronald LucA case study: Ideas on implementationNeuRona ‒ AI powered secretary2019-02-09T22:13:00+00:002019-02-09T22:13:00+00:00https://ronaldluc.com/assistant-rona<h2 id="assistant-senses-and-capabilities">Assistant senses and capabilities</h2>
<ul>
<li>Vision
<ul>
<li>A standard web camera</li>
<li>Exposition and other settings are optimized in real-time to enhance the results of face recognition</li>
</ul>
</li>
<li>Hearing
<ul>
<li>A shotgun microphone</li>
<li>Wide range of STT engines can be used, currently off-line Phonexia</li>
</ul>
</li>
<li>Voice
<ul>
<li>A speaker</li>
<li>Powered by Google TTS</li>
</ul>
</li>
<li>Email</li>
</ul>
<h2 id="face-generation">Face generation</h2>
<p>Main goal is to create a realistic video for any sound wave. The generation is split into 2 neural networks to handle the information transformation.</p>
<ol>
<li>Sound waves \(\rightarrow\) facial landmarks coordinates</li>
</ol>
<figure>
<a href="../../images/2019/02/sound2land.png"><img src="../../images/2019/02/sound2land.png" /></a>
<figcaption>Sound2landmarks</figcaption>
</figure>
<ol>
<li>Drawn landmarks \(\rightarrow\) image of a face</li>
</ol>
<figure>
<a href="../../images/2019/02/land2img.png"><img src="../../images/2019/02/land2img.png" /></a>
<figcaption>Landmarks2image</figcaption>
</figure>
<p>Resulting image of the lower face is then masked and inserted into a real image.</p>
<figure>
<a href="../../images/2019/02/img2full.png"><img src="../../images/2019/02/img2full.png" /></a>
<figcaption>Image2fullPicture</figcaption>
</figure>
<h3 id="sound2landmarks">Sound2landmarks</h3>
<p>Neural network architecture overview:</p>
<figure>
<a href="../../images/2019/02/sound2landNet.png"><img src="../../images/2019/02/sound2landNet.png" /></a>
<figcaption>Architecture: sound2mouth</figcaption>
</figure>
<ul>
<li>First try didn’t go so well (the “raw” coordinates are converted to a video)</li>
</ul>
<figure>
<a href="../../images/2019/02/sound2landFirst.gif"><img src="../../images/2019/02/sound2landFirst.gif" /></a>
<figcaption>Sound2landmarks first try</figcaption>
</figure>
<ul>
<li>It looks like a face, but the network is overfitting and gets into “idle” mode after few seconds.</li>
<li>How to fix it?
<ul>
<li>Coordinates normalization to range [0, 1] ‒ to give the network a range to operate in</li>
<li>Different lengths of sound‒video sequences ‒ to fight the output idle after few seconds</li>
<li>Dropout ‒ to fight the overfitting</li>
</ul>
</li>
</ul>
<figure>
<a href="../../images/2019/02/landmarksVid.gif"><img src="../../images/2019/02/landmarksVid.gif" /></a>
<figcaption>Sound2landmarks better</figcaption>
</figure>
<h3 id="landmarks2images">Landmarks2images</h3>
<p>First neural architecture was a conditional GAN with U-net and a landmarks image instead of random variable also known as <a href="https://ronaldluc.com/ai/Pix2pix/"><strong>pix2pix</strong></a><sup id="fnref:fa" role="doc-noteref"><a href="#fn:fa" class="footnote" rel="footnote">1</a></sup>.</p>
<figure>
<a href="../../images/2019/02/pix2pix.png"><img src="../../images/2019/02/pix2pix.png" /></a>
<figcaption>Architecture: pix2pix</figcaption>
</figure>
<figure>
<a href="../../images/2019/02/video1.gif"><img src="../../images/2019/02/video1.gif" /></a>
<figcaption>Landmarks2images first try</figcaption>
</figure>
<p>It worked, but because of the used U-net 256×256 px image is the biggest that can be trained on 8 GB of VRAM. Easy workaround is to generate just the mouth and then insert it into an actual video like was done in Obama Lip Sync<sup id="fnref:fc" role="doc-noteref"><a href="#fn:fc" class="footnote" rel="footnote">2</a></sup>. The main advantages are</p>
<ul>
<li>Generation of fullHD video</li>
<li>Easier problem to learn for the cGAN</li>
</ul>
<figure>
<a href="../../images/2019/02/video2.gif"><img src="../../images/2019/02/video2.gif" /></a>
<figcaption>Landmarks2images only mouth</figcaption>
</figure>
<p>The mouth looks realistic enough, but there are problems:</p>
<ul>
<li>Some frames are corrupted ‒ this is caused mainly by a not optimal lighting conditions</li>
</ul>
<figure>
<a href="../../images/2019/02/faceTerror.png"><img src="../../images/2019/02/faceTerror.png" /></a>
<figcaption>Sound2landmarks corrupted frame</figcaption>
</figure>
<ul>
<li>The absence of time continuity causes flickering that is apparent once inserted into a real video</li>
</ul>
<figure>
<a href="../../images/2019/02/video3.gif"><img src="../../images/2019/02/video3.gif" /></a>
<figcaption>Landmarks2images first try</figcaption>
</figure>
<p>To solve this, second neural architecture was an updated version of pix2pix from paper <a href="https://ronaldluc.com/ai/Everybody-dance-now/"><strong>Everybody Dance Now</strong></a><sup id="fnref:fb" role="doc-noteref"><a href="#fn:fb" class="footnote" rel="footnote">3</a></sup>. Generator is looking at the current landmarks frame and the last generated face frame. The Discriminator also works with pairs of landmarks and faces. The source video was changed for a one with better lighting.</p>
<figure>
<a href="../../images/2019/02/Everybody_dance_now.png"><img src="../../images/2019/02/Everybody_dance_now.png" /></a>
<figcaption>Architecture: Time dependent cGAN</figcaption>
</figure>
<p>Indeed this approach worked much better than plain pix2pix. For showcase purposes the mouth is placed into still image and normal video.</p>
<figure>
<a href="../../images/2019/02/video4.gif"><img src="../../images/2019/02/video4.gif" /></a>
<figcaption>Landmarks2images inserted into still / moving video</figcaption>
</figure>
<h3 id="conclusion">Conclusion</h3>
<p>Overall this neural pipeline is able to create reasonable and arbitrarily long videos of a target person talking given a clear speech sound. Although this setup can produce plausible results, it has limitations:</p>
<ol>
<li>At least 1 hour long video of a person talking to the camera</li>
<li>Lighting without shadows needed</li>
<li>Jaw misalignment if source video has too much head movement</li>
</ol>
<h2 id="references">References</h2>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:fa" role="doc-endnote">
<p>Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros. 2016. Image-to-Image Translation with Conditional Adversarial Networks. <a href="https://arxiv.org/pdf/1611.07004.pdf">arXiv:1611.07004</a> <a href="#fnref:fa" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fc" role="doc-endnote">
<p>Supasorn Suwajanakorn, Steven M. Seitz, Ira Kemelmacher-Shlizerman. 2017. Synthesizing Obama: Learning Lip Sync from Audio. <a href="https://grail.cs.washington.edu/projects/AudioToObama/">washinton.edu</a> <a href="#fnref:fc" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fb" role="doc-endnote">
<p>Caroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros. 2018. Everybody Dance Now. <a href="https://arxiv.org/pdf/1808.07371.pdf">arXiv:1808.07371</a> <a href="#fnref:fb" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Ronald LucSound ‒> talking videoShort text classification2019-01-08T22:08:00+00:002019-01-08T22:08:00+00:00https://ronaldluc.com/short-text-classification<p>Seeking medical aid on the internet can be deceiving. Thankfully there are sites like <a href="https://www.ulekare.cz/">ulekare.cz</a> who offer professional help. It goes like this:</p>
<ul>
<li>A patient submits his problems as a text quarry (median 100 words)</li>
<li>The request goes to one of 50 practitioners, who sorts it into one of 88 categories</li>
<li>A specialist reads the patients problems again and replies via system</li>
<li>If the problem is marked as acute a nurse calls the pacient and orders him to a doctor</li>
</ul>
<hr />
<h1 id="smart-health-hackathon-prague-2018"><a href="https://www.ceehacks.com/">Smart Health Hackathon Prague 2018</a></h1>
<p>There were 12 challenges divided into 4 categories determined by the sponsors and a main price. Our team MedAI (I as a ML programmer, Jiří Diblík as a salesman and Ondřej Korol as an iOS developer) picked a challenge:</p>
<blockquote>
<p>Design a neural network able to understand patient query and match it with the right answer using Q&A database from online medical consultation platform.</p>
</blockquote>
<p><img src="../../images/2018/12/hackathon-madai-team.png" alt="hackathon madai team" /></p>
<p>We won our category from ulekare.cz and negotiated a real deployment of our model.</p>
<h3 id="the-dataset-analysis">The dataset analysis</h3>
<ul>
<li>Over 183 000 rows of (Question, Answer, Category, icd_code).</li>
<li>Questions in both Czech and Slovak</li>
<li>Questions with grammatical errors</li>
<li>Median 100 words per question</li>
<li>Average 271 words per question</li>
</ul>
<h2 id="preprocessing">Preprocessing</h2>
<p>Preprocessing pipeline overview.</p>
<figure>
<a href="../../images/2018/12/small-text-processing-u.png"><img src="../../images/2018/12/small-text-processing-u.png" /></a>
<figcaption>Source: Own work</figcaption>
</figure>
<h3 id="translation-and-word-embeddings">Translation and Word embeddings</h3>
<p>To solve the problem with multilingual questions and grammatical mistakes, the questions are first translated to English Google translate. This also enables the use of versatile pre-trained word vectors (which I couln’t find for Czech or Slovak). For this pipeline I used Standfords GloVe (Wikipedia 2014 + Gigaword 5; 6B tokens, 400K vocab, 100d)<sup id="fnref:fa" role="doc-noteref"><a href="#fn:fa" class="footnote" rel="footnote">1</a></sup>.</p>
<p>The use of world vectors injects additional information about the language and thus makes possible even training on smaller dataset.</p>
<h3 id="crop--padding">Crop & padding</h3>
<p>From the dataset analysis I figured out that 90 % of the questions are under 280 words long => I decided to fix the word count on 280 by using cropping and padding with zeros.</p>
<p>To distinguish between zero word vector and padding a new dimension was added. The extra dimension carries information whether the word is first, last, in-between or padding.</p>
<h3 id="used-tricks">Used tricks</h3>
<ul>
<li><strong>Pareto principle</strong> applied to the classes => classifying only 18 most common classes to handle over 90 % of all questions</li>
<li><strong>Normalization of class counts</strong> by multiplying the less common (significantly decreased overfitting)</li>
</ul>
<h2 id="training">Training</h2>
<h3 id="architecture">Architecture</h3>
<p>As a standard approach in NLP<sup id="fnref:fb" role="doc-noteref"><a href="#fn:fb" class="footnote" rel="footnote">2</a></sup> I tried both LSTM and 1D CNN with different hidden layer sizes and numbers of layers as stated in the paper.</p>
<h3 id="used-trics">Used trics</h3>
<ul>
<li><strong>Dropout</strong> – decreased overfitting</li>
</ul>
<h3 id="references">References</h3>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:fa" role="doc-endnote">
<p>Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. <a href="#fnref:fa" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:fb" role="doc-endnote">
<p>Wenpeng Yin, Katharina Kann, Mo Yu and Hinrich Schutze. 2017. Comparative Study of CNN and RNN for Natural Language Processing. <a href="https://arxiv.org/abs/1702.01923">arXiv:1702.01923</a> <a href="#fnref:fb" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Ronald LucImage ‒> class