Deep learning:十五(Self-Taught Learning练习)
来源:互联网 发布:apache突然403 编辑:程序博客网 时间:2024/05/29 03:28
本文转自http://www.cnblogs.com/tornadomeet/archive/2013/03/24/2979408.html
前言:
本次实验主要是练习soft- taught learning的实现。参考的资料为网页:http://deeplearning.stanford.edu/wiki/index.php/Exercise:Self-Taught_Learning。Soft-taught leaning是用的无监督学习来学习到特征提取的参数,然后用有监督学习来训练分类器。这里分别是用的sparse autoencoder和softmax regression。实验的数据依旧是手写数字数据库MNIST Dataset.
实验基础:
从前面的知识可以知道,sparse autoencoder的输出应该是和输入数据尺寸大小一样的,且很相近,那么我们训练出的sparse autoencoder模型该怎样提取出特征向量呢?其实输入样本经过sparse code提取出特征的表达式就是隐含层的输出了,首先来看看前面的经典sparse code模型,如下图所示:
拿掉那个后面的输出层后,隐含层的值就是我们所需要的特征值了,如下图所示:
从教程中可知,在unsupervised learning中有两个观点需要特别注意,一个是self-taught learning,一个是semi-supervised learning。Self-taught learning是完全无监督的。教程中有举了个例子,很好的说明了这个问题,比如说我们需要设计一个系统来分类出轿车和摩托车。如果我们给出的训练样本图片是自然界中随便下载的(也就是说这些图片中可能有轿车和摩托车,有可能都没有,且大多数情况下是没有的),然后使用的是这些样本来特征模型的话,那么此时的方法就叫做self-taught learning。如果我们训练的样本图片都是轿车和摩托车的图片,只是我们不知道哪张图对应哪种车,也就是说没有标注,此时的方法不能叫做是严格的unsupervised feature,只能叫做是semi-supervised learning。
一些matlab函数:
numel:
比如说n = numel(A)表示返回矩阵A中元素的个数。
unique:
unique为找出向量中的非重复元素并进行排序后输出。
实验结果:
采用数字5~9的样本来进行无监督训练,采用的方法是sparse autoencoder,可以提取出这些数据的权值,权值转换成图片显示如下:
但是本次实验主要是进行0~4这5个数字的分类,虽然进行无监督训练用的是数字5~9的训练样本,这依然不会影响后面的结果。只是后面的分类器设计是用的softmax regression,所以是有监督的。最后据官网网页上的结果精度是98%,而直接用原始的像素点进行分类器的设计不仅效果要差(才96%),而且训练的速度也会变慢不少。
实验主要部分代码:
stlExercise.m:
%% CS294A/CS294W Self-taught Learning Exercise% Instructions% ------------% % This file contains code that helps you get started on the% self-taught learning. You will need to complete code in feedForwardAutoencoder.m% You will also need to have implemented sparseAutoencoderCost.m and % softmaxCost.m from previous exercises.%%% ======================================================================% STEP 0: Here we provide the relevant parameters values that will% allow your sparse autoencoder to get good filters; you do not need to % change the parameters below.inputSize = 28 * 28;numLabels = 5;hiddenSize = 200;sparsityParam = 0.1; % desired average activation of the hidden units. % (This was denoted by the Greek alphabet rho, which looks like a lower-case "p", % in the lecture notes). lambda = 3e-3; % weight decay parameter beta = 3; % weight of sparsity penalty term maxIter = 400;%% ======================================================================% STEP 1: Load data from the MNIST database%% This loads our training and test data from the MNIST database files.% We have sorted the data for you in this so that you will not have to% change it.% Load MNIST database filesmnistData = loadMNISTImages('train-images.idx3-ubyte');mnistLabels = loadMNISTLabels('train-labels.idx1-ubyte');% Set Unlabeled Set (All Images)% Simulate a Labeled and Unlabeled setlabeledSet = find(mnistLabels >= 0 & mnistLabels <= 4);unlabeledSet = find(mnistLabels >= 5);%%增加的一行代码unlabeledSet = unlabeledSet(1:end/3);numTest = round(numel(labeledSet)/2);%拿一半的样本来训练%numTrain = round(numel(labeledSet)/3);trainSet = labeledSet(1:numTrain);testSet = labeledSet(numTrain+1:2*numTrain);unlabeledData = mnistData(:, unlabeledSet);%%为什么这两句连在一起都要出错呢?% pack;trainData = mnistData(:, trainSet);trainLabels = mnistLabels(trainSet)' + 1; % Shift Labels to the Range 1-5% mnistData2 = mnistData;testData = mnistData(:, testSet);testLabels = mnistLabels(testSet)' + 1; % Shift Labels to the Range 1-5% Output Some Statisticsfprintf('# examples in unlabeled set: %d\n', size(unlabeledData, 2));fprintf('# examples in supervised training set: %d\n\n', size(trainData, 2));fprintf('# examples in supervised testing set: %d\n\n', size(testData, 2));%% ======================================================================% STEP 2: Train the sparse autoencoder% This trains the sparse autoencoder on the unlabeled training% images. % Randomly initialize the parameterstheta = initializeParameters(hiddenSize, inputSize);%% ----------------- YOUR CODE HERE ----------------------% Find opttheta by running the sparse autoencoder on% unlabeledTrainingImagesopttheta = theta; addpath minFunc/options.Method = 'lbfgs';options.maxIter = 400;options.display = 'on';[opttheta, loss] = minFunc( @(p) sparseAutoencoderLoss(p, ... inputSize, hiddenSize, ... lambda, sparsityParam, ... beta, unlabeledData), ... theta, options);%% ----------------------------------------------------- % Visualize weightsW1 = reshape(opttheta(1:hiddenSize * inputSize), hiddenSize, inputSize);display_network(W1');%%======================================================================%% STEP 3: Extract Features from the Supervised Dataset% % You need to complete the code in feedForwardAutoencoder.m so that the % following command will extract features from the data.trainFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ... trainData);testFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ... testData);%%======================================================================%% STEP 4: Train the softmax classifiersoftmaxModel = struct; %% ----------------- YOUR CODE HERE ----------------------% Use softmaxTrain.m from the previous exercise to train a multi-class% classifier. % Use lambda = 1e-4 for the weight regularization for softmaxlambda = 1e-4;inputSize = hiddenSize;numClasses = numel(unique(trainLabels));%unique为找出向量中的非重复元素并进行排序% You need to compute softmaxModel using softmaxTrain on trainFeatures and% trainLabels% You need to compute softmaxModel using softmaxTrain on trainFeatures and% trainLabelsoptions.maxIter = 100;softmaxModel = softmaxTrain(inputSize, numClasses, lambda, ... trainFeatures, trainLabels, options);%% -----------------------------------------------------%%======================================================================%% STEP 5: Testing %% ----------------- YOUR CODE HERE ----------------------% Compute Predictions on the test set (testFeatures) using softmaxPredict% and softmaxModel[pred] = softmaxPredict(softmaxModel, testFeatures);%% -----------------------------------------------------% Classification Scorefprintf('Test Accuracy: %f%%\n', 100*mean(pred(:) == testLabels(:)));% (note that we shift the labels by 1, so that digit 0 now corresponds to% label 1)%% Accuracy is the proportion of correctly classified images% The results for our implementation was:%% Accuracy: 98.3%%%
feedForwardAutoencoder.m:
function [activation] = feedForwardAutoencoder(theta, hiddenSize, visibleSize, data)% theta: trained weights from the autoencoder% visibleSize: the number of input units (probably 64) % hiddenSize: the number of hidden units (probably 25) % data: Our matrix containing the training data as columns. So, data(:,i) is the i-th training example. % We first convert theta to the (W1, W2, b1, b2) matrix/vector format, so that this % follows the notation convention of the lecture notes. W1 = reshape(theta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);b1 = theta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);%% ---------- YOUR CODE HERE --------------------------------------% Instructions: Compute the activation of the hidden layer for the Sparse Autoencoder.activation = sigmoid(W1*data+repmat(b1,[1,size(data,2)]));%-------------------------------------------------------------------end%-------------------------------------------------------------------% Here's an implementation of the sigmoid function, which you may find useful% in your computation of the costs and the gradients. This inputs a (row or% column) vector (say (z1, z2, z3)) and returns (f(z1), f(z2), f(z3)). function sigm = sigmoid(x) sigm = 1 ./ (1 + exp(-x));end
参考资料:
http://deeplearning.stanford.edu/wiki/index.php/Exercise:Self-Taught_Learning
MNIST Dataset
- Deep learning:十五(Self-Taught Learning练习)
- Deep learning:十五(Self-Taught Learning练习)
- Deep learning----------Self-taught learning
- 【Deep Learning】4、Self-Taught Learning
- Self-Taught Learning To Deep Networds
- Deep Learning by Andrew Ng --- self-taught
- Deep learning-------------Self-Taught Learning to Deep Networks
- Self-Taught Learning
- 《Deep Self-Taught Learning for Weakly Supervised Object Localization》
- UFLFL Exercise:Self-Taught Learning
- UFLDL Exercise:Self-Taught Learning
- UFLDL Exercise:Self-Taught Learning
- 【UFLDL-exercise6- Self-Taught Learning】
- UFLDL Exercise: Self-Taught Learning
- Ufldl Exercise:Self-Taught Learning
- Self-taught Learning: TransferLearning from Unlabelled Data
- 自我学习(Self-Taught Learning)
- Exercise:Self-Taught Learning 代码示例
- ARM Linux对中断的处理--中断管理系统的初始化
- C++编程学习之LeetCode OJ
- C语言的规范
- POJ 1676
- Android开发 打开文件 选择文件对话框
- Deep learning:十五(Self-Taught Learning练习)
- 隙虚拟的空
- import com.sun.image.codec.jpeg.JPEGCodec不通过 找不到包
- sqlserver数据库disable和enable所有外键
- Extjs5.0新特性
- Longest Substring Without Repeating Characters
- Linux中快捷键的使用,who命令,rm命令,ps命令,cd命令,kill命令,find命令,grep命令,tar命令(gz、tar、bz2),用户管理,vim部分配置,相关命令
- linux下c程序调用reboot函数实现直接重启
- Android 线程优先级设置方法