程序博客网 > 软件注册码

cnn文本分类 --deeplearning4j为例子

来源：互联网 发布：软件注册码 编辑：程序博客网 时间：2024/06/05 08:00

绪论

deeplearning4j提供了一个英文文本分类的例子，虽说是英文，但是中文分词之后，依葫芦画瓢用于训练，首先是利用用word2vec生成词向量模型，参考上面一篇提到的生成代码。然后是构建cnn用于文本分类。


训练数据格式：

  1. 1小额贷款 要 吗
  2. 1商家 @ 回应 十里洋场 江景会 所 订房 经理 电话 ️ 五星级 餐饮 加 五星级 k歌 包房 体验 打 8折 送 酒
  3. 1可以 啊 微信同号
  4. 1看头像 加微信
  5. 1看头像 加微信
  6. 1您 可以 加研欧 书院 微信公众号 里面 有 具体 收费 标 淮 本月 25日 书院 开展 试听 公开课 如果 您 有时间 可以来 试听
  7. 0强烈推荐 4号 4号 店长 非常 满意 朋友 发型 合适 我的 开心 因为 自然 卷 每年 得 做 头发 去过 不少 理发店 接触 不少 发型师 到 目前为止 觉得 4号 ( 帅哥 一枚 ) 最 称心如意 会 根据 发质 情况 自己的 想法 给你 设计 适合 发型 不仅 专业 而且 非常 负责 细心 我 最喜欢 细心 人 因为 只有 这样 会 做出 好的 效果 这次 先 把 发型 做出来 4号 店长 建议 染色 那样 效果 会 更好 我 不用 那么 辛苦 坐 太久 贴心 是不是 总之 满意 染色 后的 效果 更 重要 4号 亲自动手 帮 我 做 发型 强烈推荐 大家 过来 找 4号 店长 下次 来 找 你 哦 4号 店长 支持 你 djehfjdushdjd
  8. 0非常感谢 老师 给 小高 5 星 点评 想起 我 老师 一起 游览 情景 历历在目 时间 有时 像 一个 小偷 不知不觉 四月 已经 过完 2017年 已经 过去 三分之一 像 老话 连雨 不知 春 一 晴 方知夏 深 还没 来得及 好好 享受 温熏 春光 夏日 风 已 从 远处 徐徐 而来 夏天 青岛 最 美的 青岛 避暑 理想 之 地 欢迎 老师 有机会 夏天 再来 青岛 我的 手机号 即 微信号 到时 记得 微信 小高 我 来 帮 您 订 酒店 青岛 小高 祝 老师 身体健康 阖家欢乐 万事如意
  9. 0我 来 时候 他家 根本 拒绝 兑换 也是 遇 得到 气死人
  10. 0商家回应 尊敬 客人 感谢您 抽出 宝贵 时间 给 我们 评价 心瑞 国际 月子会所 来源于 台湾 拥有 26年 台式 母婴护理 经验 聚集 最 专业 最 精湛 台湾 护理 技术 团队 精心 定制 专属 护理服务 秉承 规范 操作 细致入微 精益求精 服务理念 为 产 后妈 咪 提供 从 饮食 护理 康复 早教 心理健康 等 全方位 贴身 服务模式 给 孕产 期间 家庭 全方位 专业 照护 舒适 体验 会所 紧邻 国内 最好 医院 协和医院 产后 妈咪 宝宝 都有 坚实 医疗保障 我们 免费提供 健身会所 游泳馆 给 入住 客人 家属 使用 , 我们 会 不定期 举办 丰富多彩 活动 让 更多 孕妈咪 们 了解 孕期 保健知识 新生儿 喂养 知识 哦 非常 期待 下次 妈咪 见面 哦 心瑞 国际 月子会所 全体员工 祝 服务热线 座机号码 请关注 微信号 微信账号
  11. 0商家回应 亲爱 贵宾 感谢您 襄阳 巴厘岛 休闲 度假酒店 肯定 支持 酒店 休闲会所 主要 提供 休闲 洗浴 游泳 桑拿 干湿 蒸 足疗 按摩 等 项目 如此 次 体验 没能 让 您 满意 我们 表示 深深 歉意 我们 足疗 专业 休闲 手法 所有 技师 素质 高 够 专业 各 具 魅力 服务 更好 巴厘岛 一切 以 您 最 舒适 休闲 方式 为先 您 满意 我们 继续 进步 动力 酒店 全体员工 期待 您 再次光临
  12. 0人 直接 不行 必须 每人 点 一份 主食 我 那 钱 我 给你 东西 不用 上了 吃 不掉 浪费 服务员 b 那 再 来个 茶 吧 饭 毕 服务员 c 过来 我们 要 续 杯茶 他 可能 听懂 把 bill 拿走 一会 送来 一个新 上面 加 一杯 茶 我们 收费 那 不要 他 直接 气呼呼 拿走 新 bill 回来 时候 把 新 打印 去掉 杯茶 bill 直接 摔 桌子 然后 回头 走 我 擦 你 真 牛逼 我 走 这么多 英联邦 国家 地区 别人 一看 中国 人 客客气气 何况 还是 顾客 你 妹的
  13. 0维权 群 怎么 加 我 被 套路

训练代码:

LabeledSentence.java

  1. package com.dianping.cnn.textclassify;
  3. import java.io.BufferedReader;
  4. import java.io.FileInputStream;
  5. import java.io.InputStreamReader;
  6. import java.util.ArrayList;
  7. import java.util.Collections;
  8. import java.util.HashMap;
  9. import java.util.List;
  10. import java.util.Map;
  11. import java.util.Random;
  13. import org.datavec.api.util.RandomUtils;
  14. import org.deeplearning4j.berkeley.Pair;
  15. import org.deeplearning4j.iterator.LabeledSentenceProvider;
  16. import org.nd4j.linalg.collection.CompactHeapStringList;
  18. public class LabeledSentence implements LabeledSentenceProvider {
  20. private int totalCount;
  21. private Map<String, List<String>> filesByLabel;
  22. private List<String> normList;
  23. private List<String> negList;
  24. private final List<String> sentenslist;
  25. private final int[] labelIndexes;
  26. private final Random rng;
  27. private final int[] order;
  28. private final List<String> allLabels;
  29. private int cursor = 0;
  31. public LabeledSentence(String path) {
  32. this(path, new Random());
  33. }
  35. public LabeledSentence(String path, Random rng) {
  36. totalCount = 0;
  37. filesByLabel = new HashMap<String, List<String>>();
  38. normList = new ArrayList<String>();
  39. negList = new ArrayList<>();
  40. BufferedReader buffered = null;
  41. try {
  42. buffered = new BufferedReader(new InputStreamReader(
  43. new FileInputStream(path)));
  44. String line = buffered.readLine();
  45. while (line != null) {
  47. String[] lines = line.split("\t");
  48. String label = lines[0];
  49. String contennt = lines[1];
  50. if ("1".equalsIgnoreCase(label)) {
  51. normList.add(contennt);
  52. } else if("0".equalsIgnoreCase(label)) {
  53. negList.add(contennt);
  54. }
  55. totalCount++;
  56. line = buffered.readLine();
  57. }
  58. buffered.close();
  59. } catch (Exception e) {
  60. e.printStackTrace();
  61. }
  62. System.out.println("totalCount is:"+totalCount);
  63. filesByLabel.put("1", normList);
  64. filesByLabel.put("0", negList);
  65. this.rng = rng;
  66. if (rng == null) {
  67. order = null;
  68. } else {
  69. order = new int[totalCount];
  70. for (int i = 0; i < totalCount; i++) {
  71. order[i] = i;
  72. }
  73. RandomUtils.shuffleInPlace(order, rng);
  74. }
  76. allLabels = new ArrayList<>(filesByLabel.keySet());
  77. Collections.sort(allLabels);
  79. Map<String, Integer> labelsToIdx = new HashMap<>();
  80. for (int i = 0; i < allLabels.size(); i++) {
  81. labelsToIdx.put(allLabels.get(i), i);
  82. }
  84. sentenslist = new CompactHeapStringList();
  85. labelIndexes = new int[totalCount];
  86. int position = 0;
  87. for (Map.Entry<String, List<String>> entry : filesByLabel.entrySet()) {
  88. int labelIdx = labelsToIdx.get(entry.getKey());
  89. for (String f : entry.getValue()) {
  90. sentenslist.add(f);
  91. labelIndexes[position] = labelIdx;
  92. position++;
  93. }
  94. }
  96. }
  98. @Override
  99. public boolean hasNext() {
  100. return cursor < totalCount;
  101. }
  103. @Override
  104. public Pair<String, String> nextSentence() {
  105. int idx;
  106. if (rng == null) {
  107. idx = cursor++;
  108. } else {
  109. idx = order[cursor++];
  110. }
  111. ;
  112. String label = allLabels.get(labelIndexes[idx]);
  113. String sentence;
  114. sentence = sentenslist.get(idx);
  116. return new Pair<>(sentence, label);
  117. }
  119. @Override
  120. public void reset() {
  121. cursor = 0;
  122. if (rng != null) {
  123. RandomUtils.shuffleInPlace(order, rng);
  124. }
  125. }
  127. @Override
  128. public int totalNumSentences() {
  129. return totalCount;
  130. }
  132. @Override
  133. public List<String> allLabels() {
  134. return allLabels;
  135. }
  137. @Override
  138. public int numLabelClasses() {
  139. return allLabels.size();
  140. }
  142. }

CnnSentenceDataSetIterator.java

  1. package com.dianping.cnn.textclassify;
  4. import lombok.AllArgsConstructor;
  5. import lombok.NonNull;
  7. import org.deeplearning4j.berkeley.Pair;
  8. import org.deeplearning4j.iterator.LabeledSentenceProvider;
  9. import org.deeplearning4j.iterator.provider.LabelAwareConverter;
  10. import org.deeplearning4j.models.embeddings.wordvectors.WordVectors;
  11. import org.deeplearning4j.text.documentiterator.LabelAwareDocumentIterator;
  12. import org.deeplearning4j.text.documentiterator.LabelAwareIterator;
  13. import org.deeplearning4j.text.documentiterator.interoperability.DocumentIteratorConverter;
  14. import org.deeplearning4j.text.sentenceiterator.SentenceIterator;
  15. import org.deeplearning4j.text.sentenceiterator.interoperability.SentenceIteratorConverter;
  16. import org.deeplearning4j.text.sentenceiterator.labelaware.LabelAwareSentenceIterator;
  17. import org.deeplearning4j.text.tokenization.tokenizer.Tokenizer;
  18. import org.deeplearning4j.text.tokenization.tokenizerfactory.DefaultTokenizerFactory;
  19. import org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory;
  20. import org.nd4j.linalg.api.ndarray.INDArray;
  21. import org.nd4j.linalg.dataset.DataSet;
  22. import org.nd4j.linalg.dataset.api.DataSetPreProcessor;
  23. import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
  24. import org.nd4j.linalg.factory.Nd4j;
  25. import org.nd4j.linalg.indexing.INDArrayIndex;
  26. import org.nd4j.linalg.indexing.NDArrayIndex;
  28. import java.util.*;
  29. public class CnnSentenceDataSetIterator implements DataSetIterator {
  30.     public enum UnknownWordHandling {
  31.         RemoveWord, UseUnknownVector
  32.     }
  34.     private static final String UNKNOWN_WORD_SENTINEL = "UNKNOWN_WORD_SENTINEL";
  36.     private LabeledSentenceProvider sentenceProvider = null;
  37.     private WordVectors wordVectors;
  38.     private TokenizerFactory tokenizerFactory;
  39.     private UnknownWordHandling unknownWordHandling;
  40.     private boolean useNormalizedWordVectors;
  41.     private int minibatchSize;
  42.     private int maxSentenceLength;
  43.     private boolean sentencesAlongHeight;
  44.     private DataSetPreProcessor dataSetPreProcessor;
  46.     private int wordVectorSize;
  47.     private int numClasses;
  48.     private Map<String, Integer> labelClassMap;
  49.     private INDArray unknown;
  51.     private int cursor = 0;
  53.     private CnnSentenceDataSetIterator(Builder builder) {
  54.         this.sentenceProvider = builder.sentenceProvider;
  55.         this.wordVectors = builder.wordVectors;
  56.         this.tokenizerFactory = builder.tokenizerFactory;
  57.         this.unknownWordHandling = builder.unknownWordHandling;
  58.         this.useNormalizedWordVectors = builder.useNormalizedWordVectors;
  59.         this.minibatchSize = builder.minibatchSize;
  60.         this.maxSentenceLength = builder.maxSentenceLength;
  61.         this.sentencesAlongHeight = builder.sentencesAlongHeight;
  62.         this.dataSetPreProcessor = builder.dataSetPreProcessor;
  65.         this.numClasses = this.sentenceProvider.numLabelClasses();
  66.         this.labelClassMap = new HashMap<>();
  67.         int count = 0;
  68.         //First: sort the labels to ensure the same label assignment order (say train vs. test)
  69.         List<String> sortedLabels = new ArrayList<>(this.sentenceProvider.allLabels());
  70.         Collections.sort(sortedLabels);
  72.         for (String s : sortedLabels) {
  73.             this.labelClassMap.put(s, count++);
  74.         }
  75.         if (unknownWordHandling == UnknownWordHandling.UseUnknownVector) {
  76.             if (useNormalizedWordVectors) {
  77.                 wordVectors.getWordVectorMatrixNormalized(wordVectors.getUNK());
  78.             } else {
  79.                 wordVectors.getWordVectorMatrix(wordVectors.getUNK());
  80.             }
  81.         }
  83.         this.wordVectorSize = wordVectors.getWordVector(wordVectors.vocab().wordAtIndex(0)).length;
  84.     }
  86.     /**
  87.      * Generally used post training time to load a single sentence for predictions
  88.      */
  89.     public INDArray loadSingleSentence(String sentence) {
  90.         List<String> tokens = tokenizeSentence(sentence);
  92.         int[] featuresShape = new int[] {1, 1, 0, 0};
  93.         if (sentencesAlongHeight) {
  94.             featuresShape[2] = Math.min(maxSentenceLength, tokens.size());
  95.             featuresShape[3] = wordVectorSize;
  96.         } else {
  97.             featuresShape[2] = wordVectorSize;
  98.             featuresShape[3] = Math.min(maxSentenceLength, tokens.size());
  99.         }
  101.         INDArray features = Nd4j.create(featuresShape);
  102.         int length = (sentencesAlongHeight ? featuresShape[2] : featuresShape[3]);
  103.         for (int i = 0; i < length; i++) {
  104.             INDArray vector = getVector(tokens.get(i));
  106.             INDArrayIndex[] indices = new INDArrayIndex[4];
  107.             indices[0] = NDArrayIndex.point(0);
  108.             indices[1] = NDArrayIndex.point(0);
  109.             if (sentencesAlongHeight) {
  110.                 indices[2] = NDArrayIndex.point(i);
  111.                 indices[3] = NDArrayIndex.all();
  112.             } else {
  113.                 indices[2] = NDArrayIndex.all();
  114.                 indices[3] = NDArrayIndex.point(i);
  115.             }
  117.             features.put(indices, vector);
  118.         }
  120.         return features;
  121.     }
  123.     private INDArray getVector(String word) {
  124.         INDArray vector;
  125.         if (unknownWordHandling == UnknownWordHandling.UseUnknownVector && word == UNKNOWN_WORD_SENTINEL) { //Yes, this *should* be using == for the sentinel String here
  126.             vector = unknown;
  127.         } else {
  128.             if (useNormalizedWordVectors) {
  129.                 vector = wordVectors.getWordVectorMatrixNormalized(word);
  130.             } else {
  131.                 vector = wordVectors.getWordVectorMatrix(word);
  132.             }
  133.         }
  134.         return vector;
  135.     }
  137.     private List<String> tokenizeSentence(String sentence) {
  138.         Tokenizer t = tokenizerFactory.create(sentence);
  140.         List<String> tokens = new ArrayList<>();
  141.         while (t.hasMoreTokens()) {
  142.             String token = t.nextToken();
  143.             if (!wordVectors.hasWord(token)) {
  144.                 switch (unknownWordHandling) {
  145.                     case RemoveWord:
  146.                         continue;
  147.                     case UseUnknownVector:
  148.                         token = UNKNOWN_WORD_SENTINEL;
  149.                 }
  150.             }
  151.             tokens.add(token);
  152.         }
  153.         return tokens;
  154.     }
  156.     public Map<String, Integer> getLabelClassMap() {
  157.         return new HashMap<>(labelClassMap);
  158.     }
  160.     @Override
  161.     public List<String> getLabels() {
  162.         //We don't want to just return the list from the LabelledSentenceProvider, as we sorted them earlier to do the
  163.         // String -> Integer mapping
  164.         String[] str = new String[labelClassMap.size()];
  165.         for (Map.Entry<String, Integer> e : labelClassMap.entrySet()) {
  166.             str[e.getValue()] = e.getKey();
  167.         }
  168.         return Arrays.asList(str);
  169.     }
  171.     @Override
  172.     public boolean hasNext() {
  173.         if (sentenceProvider == null) {
  174.             throw new UnsupportedOperationException("Cannot do next/hasNext without a sentence provider");
  175.         }
  176.         return sentenceProvider.hasNext();
  177.     }
  179.     @Override
  180.     public DataSet next() {
  181.         return next(minibatchSize);
  182.     }
  184.     @Override
  185.     public DataSet next(int num) {
  186.         if (sentenceProvider == null) {
  187.             throw new UnsupportedOperationException("Cannot do next/hasNext without a sentence provider");
  188.         }
  191.         List<Pair<List<String>, String>> tokenizedSentences = new ArrayList<>(num);
  192.         int maxLength = -1;
  193.         int minLength = Integer.MAX_VALUE; //Track to we know if we can skip mask creation for "all same length" case
  194.         for (int i = 0; i < num && sentenceProvider.hasNext(); i++) {
  195.             Pair<String, String> p = sentenceProvider.nextSentence();
  196.             List<String> tokens = tokenizeSentence(p.getFirst());
  198.             maxLength = Math.max(maxLength, tokens.size());
  199.             tokenizedSentences.add(new Pair<>(tokens, p.getSecond()));
  200.         }
  202.         if (maxSentenceLength > 0 && maxLength > maxSentenceLength) {
  203.             maxLength = maxSentenceLength;
  204.         }
  206.         int currMinibatchSize = tokenizedSentences.size();
  207.         INDArray labels = Nd4j.create(currMinibatchSize, numClasses);
  208.         for (int i = 0; i < tokenizedSentences.size(); i++) {
  209.             String labelStr = tokenizedSentences.get(i).getSecond();
  210.             if (!labelClassMap.containsKey(labelStr)) {
  211.                 throw new IllegalStateException("Got label \"" + labelStr
  212.                                 + "\" that is not present in list of LabeledSentenceProvider labels");
  213.             }
  215.             int labelIdx = labelClassMap.get(labelStr);
  217.             labels.putScalar(i, labelIdx, 1.0);
  218.         }
  220.         int[] featuresShape = new int[4];
  221.         featuresShape[0] = currMinibatchSize;
  222.         featuresShape[1] = 1;
  223.         if (sentencesAlongHeight) {
  224.             featuresShape[2] = maxLength;
  225.             featuresShape[3] = wordVectorSize;
  226.         } else {
  227.             featuresShape[2] = wordVectorSize;
  228.             featuresShape[3] = maxLength;
  229.         }
  231.         INDArray features = Nd4j.create(featuresShape);
  232.         for (int i = 0; i < currMinibatchSize; i++) {
  233.             List<String> currSentence = tokenizedSentences.get(i).getFirst();
  235.             for (int j = 0; j < currSentence.size() && j < maxSentenceLength; j++) {
  236.                 INDArray vector = getVector(currSentence.get(j));
  238.                 INDArrayIndex[] indices = new INDArrayIndex[4];
  239.                 //TODO REUSE
  240.                 indices[0] = NDArrayIndex.point(i);
  241.                 indices[1] = NDArrayIndex.point(0);
  242.                 if (sentencesAlongHeight) {
  243.                     indices[2] = NDArrayIndex.point(j);
  244.                     indices[3] = NDArrayIndex.all();
  245.                 } else {
  246.                     indices[2] = NDArrayIndex.all();
  247.                     indices[3] = NDArrayIndex.point(j);
  248.                 }
  250.                 features.put(indices, vector);
  251.             }
  252.         }
  254.         INDArray featuresMask = null;
  255.         if (minLength != maxLength) {
  256.             featuresMask = Nd4j.create(currMinibatchSize, maxLength);
  258.             for (int i = 0; i < currMinibatchSize; i++) {
  259.                 int sentenceLength = tokenizedSentences.get(i).getFirst().size();
  260.                 if (sentenceLength >= maxLength) {
  261.                     featuresMask.getRow(i).assign(1.0);
  262.                 } else {
  263.                     featuresMask.get(NDArrayIndex.point(i), NDArrayIndex.interval(0, sentenceLength)).assign(1.0);
  264.                 }
  265.             }
  266.         }
  268.         DataSet ds = new DataSet(features, labels, featuresMask, null);
  270.         if (dataSetPreProcessor != null) {
  271.             dataSetPreProcessor.preProcess(ds);
  272.         }
  274.         cursor += ds.numExamples();
  275.         return ds;
  276.     }
  278.     @Override
  279.     public int totalExamples() {
  280.         return sentenceProvider.totalNumSentences();
  281.     }
  283.     @Override
  284.     public int inputColumns() {
  285.         return wordVectorSize;
  286.     }
  288.     @Override
  289.     public int totalOutcomes() {
  290.         return numClasses;
  291.     }
  293.     @Override
  294.     public boolean resetSupported() {
  295.         return true;
  296.     }
  298.     @Override
  299.     public boolean asyncSupported() {
  300.         return true;
  301.     }
  303.     @Override
  304.     public void reset() {
  305.         cursor = 0;
  306.         sentenceProvider.reset();
  307.     }
  309.     @Override
  310.     public int batch() {
  311.         return minibatchSize;
  312.     }
  314.     @Override
  315.     public int cursor() {
  316.         return cursor;
  317.     }
  319.     @Override
  320.     public int numExamples() {
  321.         return totalExamples();
  322.     }
  324.     @Override
  325.     public void setPreProcessor(DataSetPreProcessor preProcessor) {
  326.         this.dataSetPreProcessor = preProcessor;
  327.     }
  329.     @Override
  330.     public DataSetPreProcessor getPreProcessor() {
  331.         return dataSetPreProcessor;
  332.     }
  334.     @Override
  335.     public void remove() {
  336.         throw new UnsupportedOperationException("Not supported");
  337.     }
  339.     public static class Builder {
  341.         private LabeledSentenceProvider sentenceProvider = null;
  342.         private WordVectors wordVectors;
  343.         private TokenizerFactory tokenizerFactory = new DefaultTokenizerFactory();
  344.         private UnknownWordHandling unknownWordHandling = UnknownWordHandling.RemoveWord;
  345.         private boolean useNormalizedWordVectors = true;
  346.         private int maxSentenceLength = -1;
  347.         private int minibatchSize = 32;
  348.         private boolean sentencesAlongHeight = true;
  349.         private DataSetPreProcessor dataSetPreProcessor;
  351.         /**
  352.          * Specify how the (labelled) sentences / documents should be provided
  353.          */
  354.         public Builder sentenceProvider(LabeledSentenceProvider labeledSentenceProvider) {
  355.             this.sentenceProvider = labeledSentenceProvider;
  356.             return this;
  357.         }
  359.         /**
  360.          * Specify how the (labelled) sentences / documents should be provided
  361.          */
  362.         public Builder sentenceProvider(LabelAwareIterator iterator, @NonNull List<String> labels) {
  363.             LabelAwareConverter converter = new LabelAwareConverter(iterator, labels);
  364.             return sentenceProvider(converter);
  365.         }
  367.         /**
  368.          * Specify how the (labelled) sentences / documents should be provided
  369.          */
  370.         public Builder sentenceProvider(LabelAwareDocumentIterator iterator, @NonNull List<String> labels) {
  371.             DocumentIteratorConverter converter = new DocumentIteratorConverter(iterator);
  372.             return sentenceProvider(converter, labels);
  373.         }
  375.         /**
  376.          * Specify how the (labelled) sentences / documents should be provided
  377.          */
  378.         public Builder sentenceProvider(LabelAwareSentenceIterator iterator, @NonNull List<String> labels) {
  379.             SentenceIteratorConverter converter = new SentenceIteratorConverter(iterator);
  380.             return sentenceProvider(converter, labels);
  381.         }
  384.         /**
  385.          * Provide the WordVectors instance that should be used for training
  386.          */
  387.         public Builder wordVectors(WordVectors wordVectors) {
  388.             this.wordVectors = wordVectors;
  389.             return this;
  390.         }
  392.         /**
  393.          * The {@link TokenizerFactory} that should be used. Defaults to {@link DefaultTokenizerFactory}
  394.          */
  395.         public Builder tokenizerFactory(TokenizerFactory tokenizerFactory) {
  396.             this.tokenizerFactory = tokenizerFactory;
  397.             return this;
  398.         }
  400.         /**
  401.          * Specify how unknown words (those that don't have a word vector in the provided WordVectors instance) should be
  402.          * handled. Default: remove/ignore unknown words.
  403.          */
  404.         public Builder unknownWordHandling(UnknownWordHandling unknownWordHandling) {
  405.             this.unknownWordHandling = unknownWordHandling;
  406.             return this;
  407.         }
  409.         /**
  410.          * Minibatch size to use for the DataSetIterator
  411.          */
  412.         public Builder minibatchSize(int minibatchSize) {
  413.             this.minibatchSize = minibatchSize;
  414.             return this;
  415.         }
  417.         /**
  418.          * Whether normalized word vectors should be used. Default: true
  419.          */
  420.         public Builder useNormalizedWordVectors(boolean useNormalizedWordVectors) {
  421.             this.useNormalizedWordVectors = useNormalizedWordVectors;
  422.             return this;
  423.         }
  425.         /**
  426.          * Maximum sentence/document length. If sentences exceed this, they will be truncated to this length by
  427.          * taking the first 'maxSentenceLength' known words.
  428.          */
  429.         public Builder maxSentenceLength(int maxSentenceLength) {
  430.             this.maxSentenceLength = maxSentenceLength;
  431.             return this;
  432.         }
  434.         /**
  435.          * If true (default): output features data with shape [minibatchSize, 1, maxSentenceLength, wordVectorSize]<br>
  436.          * If false: output features with shape [minibatchSize, 1, wordVectorSize, maxSentenceLength]
  437.          */
  438.         public Builder sentencesAlongHeight(boolean sentencesAlongHeight) {
  439.             this.sentencesAlongHeight = sentencesAlongHeight;
  440.             return this;
  441.         }
  443.         /**
  444.          * Optional DataSetPreProcessor
  445.          */
  446.         public Builder dataSetPreProcessor(DataSetPreProcessor dataSetPreProcessor) {
  447.             this.dataSetPreProcessor = dataSetPreProcessor;
  448.             return this;
  449.         }
  451.         public CnnSentenceDataSetIterator build() {
  452.             if (wordVectors == null) {
  453.                 throw new IllegalStateException(
  454.                                 "Cannot build CnnSentenceDataSetIterator without a WordVectors instance");
  455.             }
  457.             return new CnnSentenceDataSetIterator(this);
  458.         }
  460.     }
  461. }

TrainAdxCnnModel.java

  1. package com.dianping.cnn.textclassify;
  3. import java.io.File;
  4. import java.io.FileNotFoundException;
  5. import java.io.UnsupportedEncodingException;
  6. import java.util.List;
  7. import java.util.Random;
  9. import org.deeplearning4j.eval.Evaluation;
  10. import org.deeplearning4j.iterator.LabeledSentenceProvider;
  11. import org.deeplearning4j.models.embeddings.loader.WordVectorSerializer;
  12. import org.deeplearning4j.models.embeddings.wordvectors.WordVectors;
  13. import org.deeplearning4j.nn.conf.ComputationGraphConfiguration;
  14. import org.deeplearning4j.nn.conf.ConvolutionMode;
  15. import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
  16. import org.deeplearning4j.nn.conf.Updater;
  17. import org.deeplearning4j.nn.conf.graph.MergeVertex;
  18. import org.deeplearning4j.nn.conf.layers.ConvolutionLayer;
  19. import org.deeplearning4j.nn.conf.layers.GlobalPoolingLayer;
  20. import org.deeplearning4j.nn.conf.layers.OutputLayer;
  21. import org.deeplearning4j.nn.conf.layers.PoolingType;
  22. import org.deeplearning4j.nn.graph.ComputationGraph;
  23. import org.deeplearning4j.nn.weights.WeightInit;
  24. import org.deeplearning4j.optimize.listeners.ScoreIterationListener;
  25. import org.nd4j.linalg.activations.Activation;
  26. import org.nd4j.linalg.api.ndarray.INDArray;
  27. import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
  28. import org.nd4j.linalg.lossfunctions.LossFunctions;
  30. public class TrainAdxCnnModel {
  31. public static void main(String[] args) throws FileNotFoundException, UnsupportedEncodingException {
  33. String WORD_VECTORS_PATH = "adx/word2vec.model";
  34. // 基础配置
  35. int batchSize = 10;
  36. int vectorSize = 100; // 词典向量的维度,这边是100
  37. int nEpochs =3; // 迭代代数
  38. int truncateReviewsToLength = 256; // 词长大于256则抛弃
  39. int cnnLayerFeatureMaps = 100; // 卷积神经网络特征图标 / channels / CNN每层layer的深度
  40. PoolingType globalPoolingType = PoolingType.MAX;
  41. Random rng = new Random(100); // 随机抽样
  43. // 设置网络配置->我们有多个卷积层，每个带宽3,4,5的滤波器
  46.         ComputationGraphConfiguration config = new NeuralNetConfiguration.Builder()
  47.             .weightInit(WeightInit.RELU)
  48.             .activation(Activation.LEAKYRELU)
  49.             .updater(Updater.ADAM)
  50.             .convolutionMode(ConvolutionMode.Same)      //This is important so we can 'stack' the results later
  51.             .regularization(true).l2(0.0001)
  52.             .learningRate(0.01)
  53.             .graphBuilder()
  54.             .addInputs("input")
  55.             .addLayer("cnn3", new ConvolutionLayer.Builder()
  56.                 .kernelSize(3,vectorSize)
  57.                 .stride(1,vectorSize)
  58.                 .nIn(1)
  59.                 .nOut(cnnLayerFeatureMaps)
  60.                 .build(), "input")
  61.             .addLayer("cnn4", new ConvolutionLayer.Builder()
  62.                 .kernelSize(4,vectorSize)
  63.                 .stride(1,vectorSize)
  64.                 .nIn(1)
  65.                 .nOut(cnnLayerFeatureMaps)
  66.                 .build(), "input")
  67.             .addLayer("cnn5", new ConvolutionLayer.Builder()
  68.                 .kernelSize(5,vectorSize)
  69.                 .stride(1,vectorSize)
  70.                 .nIn(1)
  71.                 .nOut(cnnLayerFeatureMaps)
  72.                 .build(), "input")
  73.             .addVertex("merge", new MergeVertex(), "cnn3", "cnn4", "cnn5")      //Perform depth concatenation
  74.             .addLayer("globalPool", new GlobalPoolingLayer.Builder()
  75.                 .poolingType(globalPoolingType)
  76.                 .build(), "merge")
  77.             .addLayer("out", new OutputLayer.Builder()
  78.                 .lossFunction(LossFunctions.LossFunction.MCXENT)
  79.                 .activation(Activation.SOFTMAX)
  80.                 .nIn(3*cnnLayerFeatureMaps)
  81.                 .nOut(2)    //2 classes: positive or negative
  82.                 .build(), "globalPool")
  83.             .setOutputs("out")
  84.             .build();
  87. ComputationGraph net = new ComputationGraph(config);
  88. net.init();
  89.  net.setListeners(new ScoreIterationListener(1));
  91. // 加载向量字典并获取训练集合测试集的DataSetIterators
  92. System.out
  93. .println("Loading word vectors and creating DataSetIterators");
  94. WordVectors wordVectors = WordVectorSerializer.fromPair(WordVectorSerializer.loadTxt(new File(WORD_VECTORS_PATH)));
  96. DataSetIterator trainIter = getDataSetIterator(true, wordVectors,batchSize, truncateReviewsToLength, rng);
  97. DataSetIterator testIter = getDataSetIterator(false, wordVectors,batchSize, truncateReviewsToLength, rng);
  99. System.out.println("Starting training");
  100. for (int i = 0; i < nEpochs; i++) {
  101. net.fit(trainIter);
  102. trainIter.reset();
  104. // 进行网络演化(进化)获得网络判定参数
  105. Evaluation evaluation = net.evaluate(testIter);
  107. testIter.reset();
  108. System.out.println(evaluation.stats());
  109. }
  111. // 训练之后:加载一个句子并输出预测
  112. String contentsFirstPas = "我的 手机 是 手机号码";
  114.  INDArray featuresFirstNegative = ((CnnSentenceDataSetIterator)testIter).loadSingleSentence(contentsFirstPas);
  115.  INDArray predictionsFirstNegative = net.outputSingle(featuresFirstNegative);
  116.         List<String> labels = testIter.getLabels();
  118.         System.out.println("\n\nPredictions for first negative review:");
  119.         for( int i=0; i<labels.size(); i++ ){
  120.             System.out.println("P(" + labels.get(i) + ") = " + predictionsFirstNegative.getDouble(i));
  121.         }
  125. }
  127. private static DataSetIterator getDataSetIterator(boolean isTraining,
  128. WordVectors wordVectors, int minibatchSize, int maxSentenceLength,
  129. Random rng) {
  130. String path = isTraining ? "adx/rnnsenec.txt" : "adx/rnnsenectest.txt";
  131. LabeledSentenceProvider sentenceProvider = new LabeledSentence(path,
  132. rng);
  135. return new CnnSentenceDataSetIterator.Builder()
  136. .sentenceProvider(sentenceProvider).wordVectors(wordVectors)
  137. .minibatchSize(minibatchSize)
  138. .maxSentenceLength(maxSentenceLength)
  139. .useNormalizedWordVectors(false).build();
  140. }
  141. }


模型过程结果：

  1. Loading word vectors and creating DataSetIterators
  2. totalCount is:60
  3. totalCount is:17
  4. Starting training
  6. Examples labeled as 0 classified by model as 0: 9 times
  7. Examples labeled as 1 classified by model as 0: 2 times
  8. Examples labeled as 1 classified by model as 1: 6 times
  11. ==========================Scores========================================
  12.  Accuracy:        0.8824
  13.  Precision:       0.9091
  14.  Recall:          0.875
  15.  F1 Score:        0.8917
  16. ========================================================================
  18. Examples labeled as 0 classified by model as 0: 9 times
  19. Examples labeled as 1 classified by model as 0: 1 times
  20. Examples labeled as 1 classified by model as 1: 7 times
  23. ==========================Scores========================================
  24.  Accuracy:        0.9412
  25.  Precision:       0.95
  26.  Recall:          0.9375
  27.  F1 Score:        0.9437
  28. ========================================================================
  30. Examples labeled as 0 classified by model as 0: 9 times
  31. Examples labeled as 1 classified by model as 1: 8 times
  34. ==========================Scores========================================
  35.  Accuracy:        1
  36.  Precision:       1
  37.  Recall:          1
  38.  F1 Score:        1
  39. ========================================================================
  42. Predictions for first negative review:
  43. P(0) = 0.4453294575214386
  44. P(1) = 0.554670512676239
有问题联系我微信:  xuxu_ge
阅读全文
0 0
软件注册码 软件注册码
原创粉丝点击
热门IT博客
mac提醒事项怎么删除mac提醒事项怎么删除
云计算龙头股云计算龙头股
美国剪羊毛原理知乎
echart java 导出图片
飞思卡尔单片机怎么样
先锋播放器 mac
阿里云栖大会苏州
淘宝账户虾米音乐解绑
迅雷挖矿 软件
mysql数据库搭建
淘宝男装好卖吗
产品设计用什么软件
latex有mac版吗
软考程序员试题及答案
怎么看待网络用语
移动网络接收短信
中国高铁出口数据
贝亲淘宝旗舰店
css3动画和js动画
做淘宝客服收取保证金
热门问题 老师的惩罚 人脸识别 我在镇武司摸鱼那些年 重生之率土为王 我在大康的咸鱼生活 盘龙之生命进化 天生仙种 凡人之先天五行 春回大明朝 姑娘不必设防，我是瞎子 油布画 油布棚 油布帐篷 油布纸 床上油布 滤油布 厨房吸油布 素体 炒米饼 面包模具 饼干模具 压饼机 自动雨伞 全自动雨伞 自动伞收不回来 自动伞怎么收 自动伞怎么用 自动雨伞怎么收回去 油库爆炸 油库里 油库 伊朗油库爆炸 黄岛油库大爆炸 黄岛油库爆炸 油库设计规范 油彩画 油彩 油彩蜘蛛 儿童油彩画简单的图片 儿童油彩画图片 男模姜皓文油彩 东油房村 轿车油箱能抽油吗 抽油 电动油抽子 抽油杆 蒸鱼豉油和生抽的区别 塑料抽油器 抽油神器 隔油池清理抽粪 生抽豉油

程序博客网，程序员的互联网技术博客家园。csdn论坛精品 msdn技术资料都在这里