Monte Carlo蒙特卡洛算法经验与UCB算法与UCT算法

Monte Carlo方法举例：一个框里有100个苹果，找到其中最大的，闭上眼睛不断随机的从中拿出苹果，若新拿到的比持有的大则换一下，随着抽取苹果的次数变多拿到的苹果的大小会越来越接近最大值。此算法具体流程就不说了，说一下此算法的精度问题。一般来说此算法的优点即是可应付维数灾难。

我发现Monte Carlo方法存在以下问题：

假如在一维情况下随便找个函数给定范围是0到100，随机1000次可能很容易找到与最优值误差小于0.1以内的最优解。但是你只循环100次你的结果与最优值的平均误差会很大。可能有人会说你只要循环次数说一些就好了，但是假如你给定的范围是0到1000，你随机1000次可能和范围0到100随机100次误差是一样的。所以面对高维数情况，假如每个维数的取值范围都是0到100，那么搜索面积将有10000，所以你的随机次数也要相应的增大才会与最优值的误差在容许范围之内。

>>> Monte_C(1000,aaa,count_distant,10000000)
(array([10, 24,  0, 21, 27, 29, 26,  7,  2, 12, 22, 17,  3, 20,  4, 28, 14,
       23, 13, 18, 16, 25,  5,  1, 11,  8, 19,  6, 15,  9]), 10913.86694654037)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([27, 16,  6, 17,  0, 19,  4, 14, 12,  7, 11, 13,  8, 28, 15,  5, 21,
       29, 22, 18, 26, 24, 25, 10,  1,  9,  3,  2, 23, 20]), 11997.069466256598)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([ 8, 24, 10, 23, 27,  7,  3, 29, 22, 21, 13, 15, 26, 19, 20,  0, 16,
        2, 11, 17,  4, 18, 14,  9, 28,  5, 12,  6, 25,  1]), 11783.364741195388)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([25, 18, 14,  8, 12,  1, 21, 26, 22, 20, 19, 10,  9, 28, 23, 16,  6,
       29,  3,  2, 13,  5, 15,  7,  4, 27, 17,  0, 24, 11]), 11622.197063447451)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([22, 27, 24,  3, 18,  2, 13, 12,  1,  8,  5, 16, 23,  9, 29, 19,  4,
       28, 20, 15,  6, 25, 10, 17, 26,  7, 21, 11, 14,  0]), 11868.85957412103)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([12, 14,  5, 11, 16,  9,  7,  2, 18, 28,  4,  8, 24, 19,  1, 21, 29,
       10, 27, 26,  3, 13,  6, 22, 17, 15, 25, 23, 20,  0]), 11875.840220227323)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([ 6, 29,  0,  8,  4, 12,  7,  1, 13, 20, 28, 15, 26,  2,  9, 22, 17,
       18, 24,  3, 19, 21, 27, 16, 25,  5, 10, 11, 14, 23]), 11094.598814045059)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([20,  9, 12,  8, 25, 27,  0, 11,  7,  3,  5, 23,  6, 24, 15, 29, 26,
       18, 19, 16,  4, 17, 13, 21,  2, 14, 10, 22, 28,  1]), 11580.278041982749)
>>> Monte_C(1000,aaa,count_distant,10000000)
(array([26,  6, 23, 22, 25, 27,  2,  3, 20, 17, 10, 18, 11, 13, 12, 16,  0,
        7, 19, 29, 28, 24, 21,  9,  1,  4, 15,  5, 14,  8]), 10420.318402623658)
>>> Monte_C(10000,aaa,count_distant,10000000)
(array([ 5, 10,  4, 28, 17,  6, 21,  2, 18,  0, 12, 19, 22, 25,  9, 27, 15,
        7, 24,  3, 29, 14, 26,  1,  8, 20, 13, 16, 23, 11]), 10846.638328505325)
>>> Monte_C(10000,aaa,count_distant,10000000)
(array([15, 16,  8,  3, 29, 12, 17, 27, 24,  5, 11,  7, 18, 28, 13, 21, 26,
       22,  1,  9, 10, 25,  4, 23, 14,  6,  2,  0, 20, 19]), 10632.01091105917)
>>> Monte_C(10000,aaa,count_distant,10000000)
(array([15,  6,  8, 29, 20,  2, 10,  9, 23,  0, 13, 21, 24, 11,  1,  3, 25,
       12, 17,  4, 22, 19, 27, 14, 16, 18,  5, 26, 28,  7]), 10758.252503115953)
>>> Monte_C(10000,aaa,count_distant,10000000)
(array([25, 28, 27, 17,  9, 15, 23, 26, 16,  7, 13, 20, 12, 21, 11, 22,  3,
       19,  5, 29,  2, 14, 18,  4,  1,  6,  8, 24, 10,  0]), 10996.041672757885)
>>> Monte_C(10000,aaa,count_distant,10000000)
(array([27,  1, 17, 21, 12,  4, 23,  6, 16, 19, 22,  8, 14,  7,  5, 13, 26,
       28, 18, 15, 24, 25, 10, 29, 11,  0,  3,  2, 20,  9]), 10984.524932778528)
>>> Monte_C(100000,aaa,count_distant,10000000)
(array([22, 13,  3, 24,  7,  2, 19, 15,  1,  4, 21,  0, 16,  6, 28, 26, 23,
       14, 17, 18, 11, 29, 27, 10, 12, 25,  5,  8, 20,  9]), 9922.224417186882)
>>> Monte_C(1000000,aaa,count_distant,10000000)
(array([11, 25, 17,  7,  9,  8, 19, 28, 24, 22, 15,  0,  3, 16,  1,  2, 14,
       26, 12, 23, 18,  4, 10, 29, 20, 27, 21, 13,  6,  5]), 9786.022298979711)
>>> Monte_C(10000000,aaa,count_distant,10000000)
(array([20, 21, 13,  2, 11, 16, 28, 10, 23, 25, 15, 18,  3, 12, 19,  0,  8,
       27,  4, 26,  5,  1, 14, 22, 29, 24,  7, 17,  6,  9]), 9056.324589959093)
>>> 

以上是在30个点的情况下，第一个参数是计算次数，最后是移动距离，可以看到最后10倍的提高计算次数其解的优秀程度提高却是线性。

接下来，采用先聚类再用Monte Carlo算法来做最优路径，可以见到结果比直接用mc好的多，结果如下：

ccc1
[17, 16, 21, 19, 18, 13, 24, 12, 15, 14, 10, 26, 27, 20, 25, 28, 36, 35, 33, 32, 0, 1, 2, 7, 8, 5, 3, 6, 4, 9, 11, 22, 29, 30, 23, 31, 34]
>>> count_distant2(aaa2,ccc1)
3383.5374584184697
>>> Monte_C(1000,aaa2,count_distant,100000)
([13, 21, 4, 1, 23, 33, 29, 32, 9, 3, 6, 20, 26, 0, 2, 14, 11, 7, 22, 18, 25, 28, 36, 8, 10, 19, 16, 30, 35, 5, 17, 15, 12, 27, 31, 34, 24], 6505.110089959226)
>>> Monte_C(1000,aaa2,count_distant,100000)
([32, 36, 25, 31, 17, 35, 24, 19, 26, 9, 4, 28, 30, 5, 1, 23, 13, 15, 6, 33, 34, 14, 27, 12, 11, 8, 3, 20, 10, 0, 7, 2, 18, 21, 22, 16, 29], 6293.185200619953)
>>> Monte_C(1000,aaa2,count_distant,100000)
([35, 26, 25, 2, 10, 4, 15, 31, 33, 22, 21, 7, 11, 24, 6, 36, 30, 32, 27, 9, 12, 16, 29, 18, 8, 1, 0, 14, 20, 28, 17, 5, 3, 19, 34, 23, 13], 6293.7521504789065)
>>> Monte_C(1000,aaa2,count_distant,100000)
([13, 21, 18, 32, 29, 36, 34, 33, 16, 0, 28, 20, 11, 3, 35, 27, 19, 22, 30, 17, 4, 5, 14, 25, 24, 12, 9, 31, 1, 15, 26, 7, 8, 23, 10, 2, 6], 6506.925429425239)
>>> Monte_C(1000,aaa2,count_distant,100000)
([31, 34, 20, 26, 19, 7, 10, 25, 18, 36, 32, 35, 14, 16, 13, 33, 23, 30, 29, 17, 5, 12, 8, 24, 9, 22, 1, 0, 3, 6, 4, 21, 2, 11, 15, 27, 28], 5862.005403172394)
>>> Monte_C(1000,aaa2,count_distant,100000)
([33, 23, 21, 16, 6, 18, 9, 0, 15, 24, 22, 36, 31, 14, 3, 10, 26, 28, 13, 30, 29, 19, 35, 8, 12, 25, 20, 27, 5, 2, 7, 17, 32, 34, 11, 1, 4], 6260.558015093805)
>>> Monte_C(10000,aaa2,count_distant,100000)
([4, 7, 24, 25, 29, 19, 23, 8, 30, 11, 17, 13, 18, 22, 1, 26, 20, 2, 0, 34, 28, 14, 12, 27, 31, 21, 35, 36, 33, 32, 3, 6, 10, 9, 15, 5, 16], 5968.824917421585)

----------------------------------------------------------------------------------------------------------------------------------------------

UCB算法是蒙特卡洛算法的改进，看到过说此算法与蚁群算法原理相同，仔细一想确实是这么回事。

UCB算法在有些介绍里的评价子节点函数有问题，从论文中拿来是：

其中是ln不是log。

另外UCB算法中，当前状态下，如在某一棋局状态下你要判断怎么走，那么所有可以走的可能即是子节点。选择子节点进行simulation，如何simulation即是用Monte Carlo方法，首先选一个可以走的子节点，比如跳马，你跳马后用MC方法随机走，直到分出输赢为止，赢了就给此节点（跳马）加一分，最后运行几百上千次即可以得到所有子节点赢的大概的概率，选择概率最大的即是你要走的步。

UCB算法与Monte Carlo方法比较有什么优点？

假如有10个点，从起始点开始要求经过所有点一次最后回到起始点求最短路径。假设资源是搜索1000次，那么MC方法搜索的点序列肯定是完全随机的。而ucb算法首先列出10个子节点，这10个子节点就是下一步可以选择的所有可能，然后接下来的想法可能和最大似然法的思想有点类似，选定一个子节点，再随机走到终点，10个子节点都按此方式走一遍，那么如果一个子节点是好的的话，它肯定会有更大的概率走出一个短的距离，而一个不好的子节点必然小概率走出一个短的距离，所以我姑且就认为那个走出最短距离的节点是好的。然后我就尽量给好的节点多一些的机会来尝试他们，即是用上方的UCB1的公式评价哪个节点是好的。反正仔细想想的话UCB从平均来说肯定可以比MC方法省时省力一些。

其实从另一种角度来考虑，也可以说只是用MC算法来simiulation，也即是说用MC算法来计当作评价函数，计算这一步情况下你赢的可能性。而论文Monte-Carlo Tree Search: A New Framework for Game AI也说了，在游戏智能的应用中因为构建好的评价函数需要游戏相关知识，而且很复杂，所以现在用Monte Carlo方法来构建评价函数。

而以上两种方法一般情况下的应用只能用在搜索范围较小的地方，而UCT则是用在选择很多的情况下，比如在围棋上。

UCT算法

此类算法都有一个环节，就是play环节，什么是play环节，我的理解就是评价函数部分，通过play来得到当前路径的得分。由此可以推理出此算法并不是把整个树都列出来，而是根据计算复杂度和内存只设置一定层数的树的节点，这个问题我看到许多文章里都没有提到，或者我没注意到，反正我没看到。

所以此算法与UCB不同的即是，UCB中只计算和记录直接与跟节点相连的节点，而UCT则会多记录几层。

所以我觉得此算法的执行应该是这样：

1，从根节点有a0个子节点，分别play所有子节点

2，计算最优程度，找到最优的节点，若此节点是第一次计算，则直接play，否则向下到其子节点。重复2