Dtjytyk

I don't understand what is the purpose of the decision tree? The way I see it is, it is a series of if-else. Why don't I just use if-else instead of using a decision tree? It is because it decreases the complexity of my code?

I am still spared from calculating entropy and information gain because there are prebuilt algorithms for them where I just plug in the rules right? (Like ID3)

Why do we use it with machine learning now? Because we don't even have to come up with the rules while before we needed to? The machine learns from the training data and based on the attributes it can predict a result?

Does implementing ML in my code decrease overhead more and it makes my code less complex, more effective, faster?

The way I see it is, it is a series of if-else. Why don't I just use if-else instead of using a decision tree?

You are absolutely right. A decision tree is nothing else but a series of if-else statements. However, it is the way we interpret these statements as a tree that lets us build these rules automatically... I.e. given some input example set $(x_1, y_1), ..., (x_N, y_N)$... what is the best set of rules that describes what value $y$ has given a new input $x$? ID3 and alike lets us automatically create these rules. It is not really about the tree once built, it is about how we created it.

Apart from that one hardly ever uses a decision tree alone, the reason being precisely what you say: it is a pretty simplistic model that lacks expressiveness. However, it has one big advantage over other models: One can compute a single decision tree quite fast. That means that we can come up with algorithms that train many many decision trees (boosting, aka AdaBoost and GradientBoosting) on big datasets. These collection of (usually more than 500) of these simplistic models (called forest) can then express much more complicated shapes.

You could also imagine it like this: Given a 'nice' (i.e. continuous) but complicated function $f : [a,b] to mathbb{R}$ we could try to approximate this function using lines. If the function is complicated (like $sin(x)$ or so) then we produce a big error. However, we could combine lines in the way that we divide the interval $[a,b]$ into smaller parts $a = a_0 < a_1 < ... < a_M = b$ and on each $a_i, a_{i+1}$ we try to approximize $f|_{(a_i, a_{i+1})}$ (that is, $f$ restricted to this interval) by a line. By basic math (analysis) we can then approximate the function arbitrarily close (i.e. make an arbitrarily small error) if we take enough lines. Hence, we built up a complicated but accurate model from very simple ones. That is exactly the same idea that (for example) GradientBoosting uses: It builds a forest from very 'stupid' single decision trees.

Just adding to @Fabian Werner’s answer - do you remember doing Riemann Sums rule in an intro to integration? Well that too was a set of evenly partitioned if statements which you use to calculate the area under the function.

If you draw a 1D function and draw the partitions evenly what you will find is that in areas where the function has little gradient, neighboring partitions can be merged together without a great loss in accuracy. Equally, in partitions with high gradient adding more partitions will significantly improve the approximation.

Any set of partitions will approximate the function but some are clearly better than others.

Now, moving to CART models - we see data in the form of noisy points from this function and we are asked to approximate the function. By adding too many partitions we can overfit and essentially perform a nearest neighbor type model. To avoid this we limit the number of partitions our model can use (usually in the form of max depth and min samples per split). So now where should we place these splits? That is the question addressed by the splitting criteria. Areas with higher “complexity” should receive more splits as a rule of thumb and that is what gini, entropy, etc. endeavour to do.

Making predictions are just if-else statements but in the context of machine learning that is not where the power of the model comes from. The power comes from the model's ability to trade off over and under fit in a scalable manner and can be derived in a consistent probabilistic framework with theoretical guarantees in the limit of data. Finally, if we take a similar abstracted view of ML models we can say neural networks, kernel methods, Monte Carlo approaches and many more are simply addition and multiplication. Unfortunately, that is not a very useful view of the literature.

A decision tree is a partitioning of the problem domain in subsets, by means of conditions. It is usually implemented as cascaded if-then-elses. You can see it as a term that describes a complex decision logic.

Decision trees are neither more efficient nor more "supportive" of machine learning than logical tests. They are logical tests.

Also keep in mind that any algorithm is nothing more than a combination of arithmetic computations and tests, i.e. a (usually huge) decision tree.

For completeness, let us mention that in some contexts, such as machine learning, complex decision trees are built automatically, by algorithms. But this doesn't change their nature.