New Forum

Visit the new forum at http://godelsmarket.com/bb
Showing posts with label yahoo. Show all posts
Showing posts with label yahoo. Show all posts

Wednesday, July 25, 2012

Downloading Options Data From Yahoo!

I wanted to do some volatility surfaces using Yahoo! options data. Unfortunately they don't make it as easy to grab options data as it is to grab stock data.

Here's a very nice solution. (To prevent loss, source code follows; although all credit goes to link!)

function DataOut = Get_Yahoo_Options_Data(symbolid)
%Get_Yahoo_Options_Data get Option Chain Data from Yahoo
% Get Options Chain Data from Yahoo
% DataOut = Get_Yahoo_Options_Data(symbol)
% Inputs: Symbol name as a character String
% Output:  A structure with the following fields
%       data : A 1xN cell where N is the number of Expiries available
%       ExpDates : A 1xN cell array of Expiry Dates
%       Calls  : A 1xN cell array of Call Option data for each expiry
%       Puts  : A 1xN cell array of Put Option data
%       CPHeaders : Headers for the calls and puts option data
%       Headers: Headers for the data
%       FullOptionData : A combined cell array of DataOut.data
%       Last : Last Price
% Example:
%           DataOut = Get_Yahoo_Options_Data('LVS');
% (c)tradingwithmatlab.blogspot.com
DataOut = struct;
% Construct and read the URL from Yahoo Finance Website
urlText = urlread(['http://finance.yahoo.com/q/os?s=' symbolid]);
% Try getting the Table Data from URL Text 
TableData = getTableData();
% If Empty return
if(isempty(TableData))
    return
else
    DataOut.data{1} = TableData;
end
% Get the Expiry Date for later use
DataOut.ExpDates{1} = Get_Exp_Dates();
% Get Expiry Dates that are listed in the website to construct separate
% URLS for each month
NextExpiryURL = Get_Next_Expiry_URL();
if(isempty(NextExpiryURL))
   return
end

% Now read Option Tables of each Expiry month
for ik = 1:length(NextExpiryURL)
    urlText = urlread(NextExpiryURL{ik});
    DataOut.ExpDates{ik+1} = Get_Exp_Dates();
    DataOut.data{ik+1} = getTableData();
end
% Clean Up
% Convert the strings into numbers 
f = @(x)[x(:,1) num2cell(str2double(x(:,[2:8]))) x(:,9) num2cell(str2double(x(:,10:end)))];
DataOut.data = cellfun(f,DataOut.data,'uni',false);

goodDataIdx = (~cellfun('isempty',DataOut.data));
DataOut.data = DataOut.data(goodDataIdx );
DataOut.ExpDates = DataOut.ExpDates(goodDataIdx );
% Separate the data into Calls, Puts, Headers
DataOut.Calls = cellfun(@(x) x(:,[1 8 2:7]),DataOut.data,'uni',false);
DataOut.Puts = cellfun(@(x) x(:,[9 8 10:end]),DataOut.data,'uni',false);
DataOut.CPHeaders = {'Symbol','Strike','Last','Change','Bid','Ask','Volume','Open Int'};
DataOut.Headers = {'Symbol','Last','Change','Bid','Ask','Volume','Open Int','Strike',...
    'Symbol','Last','Change','Bid','Ask','Volume','Open Int'};
DataOut.FullOptionData = [DataOut.Headers ; cat(1,DataOut.data{:})];
% Get the Last Price
DataOut.Last = str2num(urlread(['http://download.finance.yahoo.com/d/quotes.csv?s=' symbolid '&f=l1&e=.csv']));

%% Get_Next_Expiry_URL
    function NextExpiry = Get_Next_Expiry_URL()
        % Get the start and end indices and look for a particular text
        Start = regexp(urlText,'View By Expiration:','end');
        end1 = regexp(urlText,'Return to Stacked View...','start');
        
        Data = urlText(Start:end1);
        Data=Data(2:end);
        % Trim the data
        Data=strtrim(Data);
        % Split the data into new lines
        newlines = regexp(Data, '[^\n]*', 'match');
        expr = '<(\w+).*?>.*?</\1>';
        if(isempty(newlines))
            NextExpiry = {};
            return
        end
        % Get the matches of particular expression
        [tok mat] = regexp(newlines{1}, expr, 'tokens', 'match');
        id1= regexp(mat{1},'</b>','start')-1;
        month{1} = mat{1}(4:id1);
        %Month and Next Expiries
        for j = 2:length(mat)-1
            id2 = regexp(mat{j},'">','end');
            id3 = regexp(mat{j},'</a','start');
            if(isempty(id3))
                return
            end
            month{j} = mat{j}(id2+1:id3-1);
            id4 = regexp(mat{j},'"','start');
            NextExpiry{j-1} = ['http://finance.yahoo.com' mat{j}(id4(1)+1:id4(2)-1)]; %#ok<*AGROW>
            NextExpiry{j-1} = regexprep(NextExpiry{j-1},'amp;','');
        end
        
    end
%% Get_Exp_Dates

    function ExpDates = Get_Exp_Dates()
        
        id1 = regexp(urlText,'Options Expiring','end');
        id2 = regexp((urlText(id1+1:id1+51)),'</b>','start');
        ExpDates = strtrim(urlText(id1+1:id1+1+id2-2));
        ExpDates=datestr(datenum(ExpDates,'dddd, mmmm dd,yyyy'));
    end

%% getTableData
    function out = getTableData()
        Main_Pattern = '.*?</table><table[^>]*>(.*?)</table';
        Tables = regexp(urlText, Main_Pattern, 'tokens');
        out = {};
        if(isempty(Tables))
            return
        end
        try
        for TableIdx = 1 : length(Tables)
            
            %Establish a row index
            rowind = 0;
            
            
            % Build cell aray of table data
            
                rows = regexpi(Tables{TableIdx}{:}, '<tr.*?>(.*?)</tr>', 'tokens');
                for rowsIdx = 1:numel(rows)
                    colind = 0;
                    if (isempty(regexprep(rows{rowsIdx}{1}, '<.*?>', '')))
                        continue
                    else
                        rowind = rowind + 1;
                    end
                    
                    headers = regexpi(rows{rowsIdx}{1}, '<th.*?>(.*?)</th>', 'tokens');
                    if ~isempty(headers)
                        for headersIdx = 1:numel(headers)
                            colind = colind + 1;
                            data = regexprep(headers{headersIdx}{1}, '<.*?>', '');
                            if (~strcmpi(data,'&nbsp;'))
                                out{rowind,colind} = strtrim(data);
                            end
                        end
                        continue
                    end
                    cols = regexpi(rows{rowsIdx}{1}, '<td.*?>(.*?)</td>', 'tokens');
                    for colsIdx = 1:numel(cols)
                        if(rowind==1)
                            if(isempty(cols{colsIdx}{1}))
                                continue
                            else
                                colind = colind + 1;
                            end
                        else
                            colind = colsIdx;
                        end
                        % The following code is required to get the sign
                        % of the change in Bid ask prices
                        data = regexprep(cols{colsIdx}{1}, '&nbsp;', ' ');
                        down=false;
                        % If Down is found then it is negative
                        if(~isempty(regexp(data,'"Down"', 'once')))
                            down=true;
                        end
                        data = regexprep(data, '<.*?>', '');
                        if(down)
                            data = ['-' strtrim(data)];
                        end
                        if (~isempty(data))
                            out{rowind,colind} = strtrim(data) ;
                        end
                    end % colsIdx
                end
                
                
        end
        out = out(3:end,:);
        catch %M  %#ok<CTCH> This depends on which version of matlab you are using
               %M.stack
        end
    end
end

Wednesday, July 11, 2012

Matlab Stock Correlation Matrix

If you enjoy the following, consider signing up for the Gödel's Market Newsletter.

So, maybe you used the Python Yahoo! data importer script I posted; or, maybe, you have your own price data. And, perhaps you want to run a script to see what the correlation is among the stocks in your database. This code will do just that.

You need to download the MySQL JDBC connector and go through the hassle of adding the .jar file to your Matlab java path. It's worth the struggle, though.

Here's the code with some comments to hopefully help you along (and remind me what I did later):

   
 %test variables to limit matrix sizes...  
 num_stocks = 500;  
 num_close_values = 20;  
 num_log_values = 19;  
   
 %connect to the database  
 conn = database('stocks','root','*****','com.mysql.jdbc.Driver','jdbc:mysql://localhost:3306/stocks');  
   
 %query the symbols in the database  
 str_query_symbols = 'SELECT distinct symbol FROM stock_prices_day';  
 symbol_cell = fetch(conn, str_query_symbols);  
   
 %create matrices to prevent matrix resizing when inserting cell values..  
 mat_log_change = zeros(num_log_values,num_stocks);  
 mat_values_adj_close = zeros(num_close_values,num_stocks);  
   
 %grab adj_close info for a each symbol; then cell2mat it into a larger  
 %matrix  
 for i = 1:num_stocks  
   symbol = cell2mat(symbol_cell(i));  
   str_query_adj_close = ['SELECT adj_close FROM stock_prices_day WHERE symbol = ''' symbol ''' and date > ''2012-06-01'''];  
   mat_values_adj_close(:,i) = cell2mat(fetch(conn, str_query_adj_close));  
     
   %use this to calculate natrual log changes as you move along  
   for j = 1: num_log_values  
     %mat_values_adj_close(j,i)  
     mat_log_change(j,i) = log((mat_values_adj_close(j,i))/(mat_values_adj_close((j+1),i)));  
   end  
 end  
   
 %next calculate correlation coefficients among the various columns  
 mat_correlations = zeros(num_stocks,num_stocks);  
 for i = 1:num_stocks  
   for j = 1:num_stocks  
     A = mat_log_change(:,i);  
     B = mat_log_change(:,j);  
     R = corrcoef(A,B);  
     mat_correlations(i,j) = R(2);  
   end  
 end  
   
 mat_correlations  

(If you've enjoyed this article, consider signing up for the Gödel's Market Newsletter.)

Tuesday, July 10, 2012

Simple Version of Yahoo! MySQL Importer

If you enjoy the following, consider signing up for the Gödel's Market Newsletter.

I decided to post the source code for my Yahoo! Stock Quote MySQL Importer Python script.

This is the simple version without the ability to add multiple stock tickers at once and no choice of time limits. Also, it doesn't download economic info or keep track of dividends.

In order to use this, you must have Python 2.7 (or something that can run the code equivalently; I don't think Python 3+ works), the Python-MySQL (MySQLdb) module, and MySQL installed (I'm using server 5.5).

Within your MySQL server you must have a "schema" (or "database") called "stocks". Actually, you can call it whatever you want, because you have the option to change the name in the Python program. But, it is preselected for "stocks".

Within this "schema"/"database" you must have a table called "stock_prices_day". This is hardcoded, but can obviously be changed if you know a little python, or just read through the code and look for any instance of "stock_prices_day" and replace them with whatever you want your table to be called.

Within "stock_prices_day" you must have the following columns with the indicated data types: "symbol" varchar(5), "date" date, "open" decimal(5,3), "high" decimal(5,3), "low" decimal(5,3), "close" decimal(5,3), "volume" int, "adj_close" decimal(5,3).

When you have all of that, you can run the following code, insert your server information (ip, username, password, etc) and the symbol you want to insert. It'll download it and insert the symbol into the database.

BAM! you're on your way to a real hedge fund-esque setup. :D (well, almost).

Rules on the code:
1) Only for personal use by investors with less than $100k in liquid assets (not for use by businesses, corporations, etc).
2) Not for resale and reuse requires my consent.
3) Post a comment if you'd like access to a more complete version (date selection, multiple symbols, etc).


 from time import sleep, strftime, localtime  
 from Tkinter import *  
 import MySQLdb  
 import urllib  
 import csv  
 import os  
   
 class App:  
   
   def __init__(self, master):  
   
     frame = Frame(master)  
     frame.pack()  
   
     self.mysqlinfo_label = Label(frame, text='MySQL fields:')  
     self.mysqlinfo_label.grid(row=0)  
   
     self.label_host = Label(frame, text='Host:')  
     self.label_host.grid(row=1)  
   
     host_text = StringVar()  
     host_text.set("127.0.0.1")  
   
     self.entry_host = Entry(frame, textvariable=host_text)  
     self.entry_host.grid(row=1, column=1)  
   
     self.label_user = Label(frame, text='User:')  
     self.label_user.grid(row=2)  
   
     user_text = StringVar()  
     user_text.set("root")  
   
     self.entry_user = Entry(frame, textvariable=user_text)  
     self.entry_user.grid(row=2, column=1)  
   
     self.label_password = Label(frame, text='Password:')  
     self.label_password.grid(row=3)  
   
     self.entry_password = Entry(frame, show="*")  
     self.entry_password.grid(row=3, column=1)  
   
     self.label_database = Label(frame, text='Database:')  
     self.label_database.grid(row=4)  
   
     database_text = StringVar()  
     database_text.set("stocks")  
   
     self.entry_database = Entry(frame, textvariable=database_text)  
     self.entry_database.grid(row=4, column=1)  
   
     self.label_empty = Label(frame, text='')  
     self.label_empty.grid(row=5)   
   
     self.label_twscontractinfo = Label(frame, text='Yahoo info:')  
     self.label_twscontractinfo.grid(row=9)  
   
     self.label_symbol = Label(frame, text='Symbol:')  
     self.label_symbol.grid(row=10)  
   
     self.entry_symbol = Entry(frame)  
     self.entry_symbol.grid(row=10, column=1)  
       
   
     self.label_empty = Label(frame, text='')  
     self.label_empty.grid(row=13)      
   
     self.button_download = Button(frame, text="Download", command=self.tws_connect)  
     self.button_download.grid(row=14, column=1)  
   
     self.button_import = Button(frame, text="Import", command=self.mysql_connect)  
     self.button_import.grid(row=15, column=1)  
   
     self.label_empty = Label(frame, text='')  
     self.label_empty.grid(row=16)   
   
   
   def tws_connect(self):  
     print "downloading csv file from yahoo..."  
     webFile = urllib.urlopen("http://ichart.finance.yahoo.com/table.csv?s=" + self.entry_symbol.get())  
     fileName = self.entry_symbol.get() + ".csv"  
     localFile = open(fileName.split('/')[-1], 'w')  
     localFile.write(webFile.read())  
     webFile.close()  
     localFile.close()  
   
     fieldnames = ['Symbol', 'Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']  
     fileName2 = self.entry_symbol.get() + "2.csv"  
     with open(fileName, 'rb') as csvinput:  
       with open(fileName2, 'wb') as csvoutput:  
         csvwriter = csv.DictWriter(csvoutput, fieldnames, delimiter=',')  
         csvwriter.writeheader()  
         for row in csv.DictReader(csvinput):  
           row['Symbol'] = self.entry_symbol.get()  
           csvwriter.writerow(row)  
   
   
   def mysql_connect(self):  
     print "connecting to mysql..."  
       
     print "MySQL host: " + self.entry_host.get()  
     print "MySQL user: " + self.entry_user.get()  
     print "MySQL database: " + self.entry_database.get()  
   
     db = MySQLdb.connect(host=self.entry_host.get(), user=self.entry_user.get(),passwd=self.entry_password.get(), db=self.entry_database.get())  
     cur = db.cursor()  
   
     current_path = os.path()  
   
     print current_path  
   
     cur.execute("load data local infile '" + current_path + self.entry_symbol.get() + "2.csv' into table `stocks`.`stock_prices_day` fields terminated by ',' lines terminated by '\n' ignore 1 lines (`symbol`,`date`,`open`,`high`,`low`,`close`,`volume`,`adj_close`);")  
     db.commit()  
     self.contract_info()  
   
   def contract_info(self):  
     print "contract info..."  
     print "Symbol: " + self.entry_symbol.get()  
   
 root = Tk()  
 root.title('Historical Data: Download and Import')  
 app = App(root)  
   
 root.mainloop()  


(If you've enjoyed this, consider signing up for the Gödel's Market Newsletter.)

Sunday, July 8, 2012

Yahoo!

So, I still haven't gotten around to packing everything into a self-installing exe. I've been extremely busy with job interviews, sharpening my programming skills, and hanging out with friends before I most likely move some 400 miles away.

I have been able to use the MySQL server I setup on a new very inexpensive server. I've downloaded Yahoo! data on the companies currently on the S&P500 and have ran several back tests with one of Charles D. Kirkpatrick's books in mind. I don't have any definitive results yet. But it looks promising.

I'm thinking of rewriting the Yahoo! program in Java for easier distribution. Or maybe C/C++. I don't know. I know I'm not ready to release it open source.

More soon.

Saturday, June 30, 2012

Yahoo! Import Update

If you enjoy the following, consider signing up for the Gödel's Market Newsletter.

I still haven't wrapped everything up into an installer. Here, however, is an up-to-date screenshot. I don't think these features will be available in the free version, but I added:

- Multiple symbols (separated by " ,") imported at once
- A way to choose how far back the data goes (jan = 00...dec = 11) so that you can update daily without inserting repeat data (I'll probably add something to make sure you're not adding repeats later).


That's it for the moment. I do want to add a txt file loader (with symbol info) and a way to capture dividend data. Perhaps I'll make one for economic and earnings data too.

(If you've enjoyed this, consider signing up for the Gödel's Market Newsletter.)

Import Yahoo Stock CSV Files into MySQL -- Program

If you enjoy this article, consider signing up for the Gödel's Market Newsletter.

This is a very preliminary version (Windows 7 tested) of what I intend to create. But, I thought I would share it.

What the program does:
- Downloads the CSV file of the specified stock symbol from Yahoo! Finance.
- Adds a column in the CSV file with symbol info so that you can insert multiple stock tickers in the same table and sort them by "Symbol."
- Connects to the selected database and inserts the data into a table called "stock_prices_day" with column values "Symbol", "Date", "Open", "High", "Low", "Close", "Volume", "Adj_Close".

-There are simple modifications that may need to be made in order for this to run on machines besides my own. I'll try to get around to it as soon as I find the errors.
- There are more complicated modifications that would obviously be nice, which I will (and have) work on. These will likely be kept to myself, unless there is substantial interest. Donations would be nice given that even the simple version took me quite some time to put together... :D (plus the benefits of this program to a quantifiable trading setup are large).

License Requirements:
- This is for personal use only.
- I do not guarantee it will work.
- I don't believe it will damage anything you own, but it possibly could.
- It gives no investment advice; it is merely a tool.
- It does not come with source.

If you would like the source, I am willing to talk. I'm going to update it somewhat, because it will probably malfunction on a system that isn't my own (hard written directories, perhaps, and dll file requirements).

Here is a screenshot followed by the download.


Download - Sorry, I have to find somewhere to upload the file...I guess blogger doesn't have anything built in. (Try This)  I'm going to have to build this into an installer...hmm. 

(If you've enjoyed this article, consider signing up for the Gödel's Market Newsletter.)