解决上位词问题

来源：互联网发布：radium for mac 编辑：程序博客网时间：2024/06/10 15:42

前几日参加了一个比赛，比赛的结果还算不错，但是有一个问题一直没有解决，赛题希望开发一款调动志愿者进行深度学习训练样本标注的APP，由于我们的解决方案开放自定义输入，例如“金毛犬”和“狗”这样的标签，同一张图片在不同的深度学习任务中应该有不同的标签（例如2017年百度西交大数据比赛的狗分类和简单的猫狗分类），这两者标签都应该保留，但是如何让机器挖掘出这两个标签的上位词关系（比赛之前我一直不知道这叫上位词关系，一直以为这叫名词从属关系）。当时的解决方案是用Word2Vec，但是由于语料较少再加上本身Word2Vec的算法特性对于上位词并不友好，所以效果很差。这几天看了看ImageNet比赛的起源，看到了一个叫WordNet的词典，联想到了比赛的问题并解决之。WordNet不做介绍，网上可以直接搜索到。考虑到数据问题，我选择用ImageNet1K映射到Pascal VOC2007，即从ImageNet中寻找在Pascal VOC中的上位词。算法本身并不复杂，主要包括广度优先搜索，由于Python队列不能判断重复元素，构建链表模并重写了相关的方法。

链表的代码如下：

class Node():    __slots__ = ['_item','_next']    def __init__(self, item):        self._item = item        self._next = None    def getItem(self):        return self._item    def getNext(self):        return self._next    def setNext(self, newnext):        self._next = newnextclass list():    def __init__(self):        self._head = None        self._size = 0    def is_Empty(self):        return self._size == 0    def append(self, item):        temp = Node(item)        if self.is_Empty():            self._head = temp        else:            current = self._head            if current.getItem() == item:                return            while current.getNext()!=None:                current = current.getNext()                if current.getItem() == item:                    return            current.setNext(temp)        self._size += 1    def pop(self):        assert not self.is_Empty()        temp = self._head        self._head = self._head.getNext()        self._size -= 1        return temp.getItem()

Python Queue.Queue的源码：

# Override these methods to implement other queue organizations# (e.g. stack or priority queue).# These will only be called with appropriate locks held # Initialize the queue representationdef _init(self, maxsize):    self.queue = deque() def _qsize(self, len=len):    return len(self.queue) # Put a new item in the queuedef _put(self, item):    self.queue.append(item)  # Get an item from the queuedef _get(self):    return self.queue.popleft()

重写后的代码：

class NewQueue(Queue.Queue):    def _init(self, maxsize=0):        self.queue = list()    def _put(self, item):        self.queue.append(item)    def _get(self):        return self.queue.pop()    def _qsize(self, len=len):        # print self.queue._size        return self.queue._size

ImageNet 1K（synset_words.txt,网上可以下载到）中的数据集如下所示（部分）：

n01440764 tench, Tinca tincan01443537 goldfish, Carassius auratusn01484850 great white shark, white shark, man-eater, man-eating shark, Carcharodon carchariasn01491361 tiger shark, Galeocerdo cuvierin01494475 hammerhead, hammerhead sharkn01496331 electric ray, crampfish, numbfish, torpedon01498041 stingrayn01514668 cockn01514859 henn01518878 ostrich, Struthio camelusn01530575 brambling, Fringilla montifringillan01531178 goldfinch, Carduelis carduelisn01532829 house finch, linnet, Carpodacus mexicanusn01534433 junco, snowbirdn01537544 indigo bunting, indigo finch, indigo bird, Passerina cyanean01558993 robin, American robin, Turdus migratorius

由于ImageNet中的词都是WordNet中的，所以不用担心在WordNet中找不到，但是需要注意的是例如great white shark在WordNet中表现的是‘great_white_shark’，所以需要操作一波。。。

另外每一个单词在WordNet中都是一个三元组，例如‘dog.n.1'，’dog‘代表词本身，’n'代表词性，‘1’代表实际含义。synset.hypernyms()可以获取当前词的上一层的上位词，所以如果需要找到你想要的上位词（本例而言就是Pascal VOC中的）。

全部代码：

# encoding:utf-8from nltk.corpus import wordnet as wnimport QueueCLASSES = ['aeroplane', 'bicycle', 'bird', 'boat',           'bottle', 'bus', 'car', 'cat', 'chair',           'cow', 'diningtable', 'dog', 'horse',           'motorbike', 'person', 'pottedplant',           'sheep', 'sofa', 'train', 'tvmonitor']file = open('synset_words.txt', 'r')file1 = open('map.txt', 'w')class Node():    __slots__ = ['_item','_next']    def __init__(self, item):        self._item = item        self._next = None    def getItem(self):        return self._item    def getNext(self):        return self._next    def setNext(self, newnext):        self._next = newnextclass list():    def __init__(self):        self._head = None        self._size = 0    def is_Empty(self):        return self._size == 0    def append(self, item):        temp = Node(item)        if self.is_Empty():            self._head = temp        else:            current = self._head            if current.getItem() == item:                return            while current.getNext()!=None:                current = current.getNext()                if current.getItem() == item:                    return            current.setNext(temp)        self._size += 1    def pop(self):        assert not self.is_Empty()        temp = self._head        self._head = self._head.getNext()        self._size -= 1        return temp.getItem()class NewQueue(Queue.Queue):    def _init(self, maxsize=0):        self.queue = list()    def _put(self, item):        self.queue.append(item)    def _get(self):        return self.queue.pop()    def _qsize(self, len=len):        # print self.queue._size        return self.queue._sizedef is_mapping(word_name, num):    q = NewQueue()    word_list = wn.synsets(word_name)    # 获取当前单词的同义词集, 并将其放置在队列中    for index in word_list:        q.put(index)    while not q.empty():        word_synset = q.get()        # print word_synset        word_name = word_synset.name()        word_name = word_name.lower()        if word_name.split('.')[0] in CLASSES:            file1.write(num + ' ' + word_name.split('.')[0] + '\n')            break        else:            word_set = wn.synset(word_name)            word_hyperlist = word_set.hypernyms()            for index in word_hyperlist:                q.put(index)if __name__ == "__main__":    for line in file:        global word        # print line        line = line.strip('\n')        num = line[:9]        line = line[10:]        if line.find(',') == -1:            word = line        else:            pos = line.find(',')            word = line[:pos]        word = word.replace(' ','_')        is_mapping(word, num)

结果展示（部分）：

n01494475 personn01514668 birdn01514859 birdn01518878 personn01530575 birdn01531178 bird

阅读全文

0 0