C4.5算法

C4.5算法是对ID3算法的改进，在决策树的生成过程中，使用了信息增益率作为属性选择的方法，其具体的算法步骤如下：

输入：训练数据集D，特征集A，阈值e
输出：决策树T
1.如果D中所有实例属于同一类C，则置T为单结点树，并将C作为该结点的类，返回T
2.如果A=∅，则置T为单结点树，并将D中实例数最大的类C作为该结点的类，返回T
3.否则，计算A中各特征对D的信息增益率，选择信息增益率最大的特征Ak
4.如果Ak的信息增益率小于阈值e，则置T为单结点树，并将D中实例数最大的类C作为该结点的类，返回T
5.否则，对Ak的每一个可能值ai，依Ak=ai将D分割为子集若干非空Di，将属性Ak作为一个结点，其每个属性值ai作为一个分支，分别构建子结点，由结点及其子结点构成树T，返回T
6.对结点i，以Di为训练集，以A−{Ak}为特征集，递归地调用步骤(1)∼(5)得到子树Ti，返回Ti

通过上述算法步骤可以发现，ID3算法和C4.5算法步骤基本一致，唯一的变化就是，在第四步时将ID3算法中的信息增益，改成了C4.5算法中的信息增益率。其他步骤两种算法完全一致。

代码实现

# 加载数据def loadDataSet(dataPath):      dataset = []      with open(dataPath) as file:          lines = file.readlines()          for line in lines:              values = line.strip().split(' ')              dataset.append(values)      return dataset  # 根据属性值，分割数据集def splitDataSet(dataset, axis, value):      retDataSet = []      for featVec in dataset:          if featVec[axis] == value:              reducedFeatVec = featVec[:axis]              reducedFeatVec.extend(featVec[axis+1:])              retDataSet.append(reducedFeatVec)      return retDataSet  # 计算数据集的信息熵def calShannonEnt(dataset):      numEntries = len(dataset) * 1.0      labelCounts = dict()      for featVec in dataset:          currentLabel = featVec[-1]          if currentLabel not in labelCounts.keys():              labelCounts[currentLabel] = 0          labelCounts[currentLabel] += 1      shannonEnt = 0.0      for key in labelCounts:          prob = labelCounts[key] / numEntries          import math          shannonEnt -= prob * math.log(prob, 2)      return shannonEnt  # 计算分割后的数据集相较于原数据集的信息增益def InfoGain(dataset, axis, baseShannonEnt):      featList = [example[axis] for example in dataset]      uniqueVals = set(featList)      newShannonEnt = 0.0      numEntries = len(dataset) * 1.0      for value in uniqueVals:          subDataSet = splitDataSet(dataset, axis, value)          ent = calShannonEnt(subDataSet)          prob = len(subDataSet) / numEntries          newShannonEnt += prob * ent      infoGain = baseShannonEnt - newShannonEnt      return infoGain  # 计算属性的分裂信息值def SplitInfo(dataset, axis):      numEntries = len(dataset) * 1.0      labelsCount = dict()      ent = 0.0      for featVec in dataset:          value = featVec[axis]          if value not in labelsCount:              labelsCount[value] = 0          labelsCount[value] += 1      for key in labelsCount:          prob = labelsCount[key] / numEntries          import math          ent -= prob * math.log(prob, 2)      return ent  # 计算属性的信息增益率def GainRate(dataset, baseset, axis, baseShannonEnt):      infoGain = InfoGain(dataset, axis, baseShannonEnt)      splitInfo = SplitInfo(baseset, axis)      return infoGain / splitInfo# 根据信息增益率，来选择属性def ChooseBestFeatureByGainRate(dataset, baseset):      numFeature = len(dataset[0]) - 1      baseShannonEnt = calShannonEnt(dataset)      bestGainRate = 0.0      bestFeature = -1      for i in range(numFeature):          gainRate = GainRate(dataset, baseset, i, baseShannonEnt)          if gainRate > bestGainRate:              bestGainRate = gainRate              bestFeature = i      return bestFeature# 构建决策树def createTree(dataset, baseset, labels):      classList = [example[-1] for example in dataset]      if classList.count(classList[0]) == len(classList):          return classList[0]      if len(dataset[0]) == 1:          return majorityCnt(classList)      bestFeature = ChooseBestFeatureByGainRate(dataset, baseset)      bestFeatureLabel = labels[bestFeature]      myTree = {bestFeatureLabel:{}}      del(labels[bestFeature])      featValues = [example[bestFeature] for example in dataset]      uniqueVals = set(featValues)      for value in uniqueVals:          subLabels = labels[:]          myTree[bestFeatureLabel][value] = \         createTree(splitDataSet(dataset, bestFeature, value), baseset, subLabels)      return myTree