Archive

Archive for June, 2008

Bypass Proxy with a Perl and Firefox Hack

June 25th, 2008

So being a developer, I rely on both my smarts and the smarts of others to get the job done.  One of the greatest ways to come up with a solution is using google to search to see how others have done similar things with code, and then crafting your own solution. The problem is many companies block blogspot, wordpress and other blogging urls thus making it impossible for you to access some of the very useful blogs out there. Luckily I run my blog off my own URL so its probably not blocked.

Well after enough frustration, I decided it was time to fix this problem. I spend 10-12hrs a day at the office, I don’t want to wait to get home to figure do my research.

I found a cgi script that acts as an http or ftp proxy from www.jmarshall.com/tools/cgiproxy/

Because I have a couple of websites up and running I had a readily available cgi-bin directory. I simply dropped the script on my server and WA-LA! I was given a form field which acted as an address bar. If I copy and pasted a blog link into this field, I was taken to the site! AWESOME!

I must mention one more thing. It actually wasn’t quite that simple. Websense was very smart at my company, and probably is at yours as well. When I first ran it, it said “Proxy Avoidence Detected!!”.

Ouch!!!!!!

Well, I obviously was not completely out of luck or I would not be writing this post…now would I?

The very simple solution which popped in my head within about 1 second of getting this error was……

You got it! Run in on a secure channel. HTTPS. This will require a certificate, but this I also already had with my ISP. So all I had to do was forward to HTTPS!

GREAT!!! I’m finally up an running!

After about a week of copying the links I found, Then visiting my https site, then pasting to the input box….I realized this was just too many steps. Ideally I’d be able to right click a link and say “OPEN WITH PROXY HACK” (and that is my next project). However, this was going to require me to learn to write a firefox extension (which i have not done yet).

So I opted for the very simple solution of adding it to my search engine drop down list on the upper right hand corner of Firefox.

Now I simply copy the link and paste it up in my search box and it forwards me to the site.

Here is the code for the FireFox Search Engine….You will just need to..
1) Edit the links.
2) Save the file as ProxyHack.xml (or whatever you’d like it to be.xml)
3) Copy the file to Path to Firefox\searchplugins
4) Restart Firefox!
5) Send me a thank you :)

Greg Perl

Perl Stochastic Math::Business::Stochastic

June 20th, 2008

I recently found a bug in the Perl Stochastic calculator which I have amended for myself and am now sharing with the world.

%D should be an xDay moving average of %K and SD should be the Moving average of %D. I have added 2 subroutines that make use of Math::Business::SMA.

I keep an array of %K values and pass it to my intializeSMA subroutine. This creates a SMA for %K based on $this->{d}. I do the same for SD. I keep an array of %D and then

keep calculate a SMA based on SD $this->{sd}.

Here is the updated Perl Module:

package Math::Business::Stochastic;

use strict;
use warnings;
use diagnostics;

our $VERSION = '0.03';
our ($kkk,@curK,@curD);
use Carp;
use List::Util qw(max min sum);

1;

sub new {
    bless {
        val  => [],
        days => 0,
    };

}

sub set_days {
    my $this = shift;
    my ($k,$d,$sd) = @_;

    croak "k must be a positive non-zero integers"  if int($k) <= 0;
    croak "d must be a positive non-zero integers"  if int($d) <= 0;
    croak "sd must be a positive non-zero integers" if int($sd) <= 0;

    $this->{k} = int($k);
    $this->{d} = int($d);
    $this->{sd} = int($sd);
    $this->{days} = int($k) + int($d) + int($sd) - 2;
	print "DAYS = " . $this->{days} . "\n";
}

sub query_k  { my $this = shift; return $this->{cur_k}; }
sub query_d  { my $this = shift; return $this->{cur_d}; }
sub query_sd { my $this = shift; return $this->{cur_sd}; }

sub insert {
    my $this = shift;
    my ($high,$low,$close) = @_;

    croak "You must set the number of days before you try to insert" if not $this->{days};
    croak "You must specify the high,low,close values" unless defined $close;
    croak "High value must be higher than low value" unless $high >= $low;
    croak "Low value must be lower than close value" unless $low <= $close;
    croak "High value must be higher than close value" unless $high >= $close;

    push @{ $this->{val_high} }, $high;
    push @{ $this->{val_low} }, $low;
    push @{ $this->{val} }, $close;

    $this->recalc;
}

sub start_with {
    my $this        = shift;
       $this->{val_high} = shift;
       $this->{val_low}  = shift;
       $this->{val}      = shift;

    croak "bad arg to start_with" unless ref($this->{val_high}) eq "ARRAY";
    croak "bad arg to start_with" unless ref($this->{val_low}) eq "ARRAY";
    croak "bad arg to start_with" unless ref($this->{val}) eq "ARRAY";
    croak "bad arg to start_with" unless @{$this->{val_high}} == @{$this->{val}};
    croak "bad arg to start_with" unless @{$this->{val_low}} == @{$this->{val}};

    $this->recalc;
}

sub recalc {
    my $this = shift;
    shift @{ $this->{val_high} } while @{ $this->{val_high} } > $this->{days};
    shift @{ $this->{val_low} } while @{ $this->{val_low} } > $this->{days};
    shift @{ $this->{val} } while @{ $this->{val} } > $this->{days};

    if( $this->{k} <= @{ $this->{val} }  ) {

        push @{ $this->{val_max} }, max( picklast($this->{k},@{$this->{val_high}}) );
        push @{ $this->{val_min} }, min( picklast($this->{k},@{$this->{val_low}}) );
        push @{ $this->{val_close_minus_min} }, $this->{val}->[-1] - $this->{val_min}->[-1];
        push @{ $this->{val_max_minus_min} }, $this->{val_max}->[-1] - $this->{val_min}->[-1];
        shift @{ $this->{val_max} } while @{ $this->{val_max} } > $this->{k};
        shift @{ $this->{val_min} } while @{ $this->{val_min} } > $this->{k};
        shift @{ $this->{val_close_minus_min} } while @{ $this->{val_close_minus_min} } > $this->{k};
        shift @{ $this->{val_max_minus_min} } while @{ $this->{val_max_minus_min} } > $this->{k};
    }
    if( $this->{k}+$this->{d}-1 <= @{ $this->{val} } ) {

	   push @{ $this->{val_d} }, sum(picklast($this->{d},@{$this->{val_close_minus_min}})) / sum(picklast($this->{d},@{$this->{val_max_minus_min}})) * 100;
        shift @{ $this->{val_d} } while @{ $this->{val_d} } > $this->{sd};
    }

    if( not defined $this->{val_max_minus_min}->[-1] or $this->{val_max_minus_min}->[-1] > 0 ) {

		if( @{ $this->{val} } == $this->{days} ) {
            $this->{cur_k}  = ($this->{val}->[-1] - $this->{val_min}->[-1]) / ($this->{val_max_minus_min}->[-1]) * 100;
			###This was added 6/20/2008 Keeps an array of %K and calculates %D as a SMA of %K then calculates Slow Stochastics using A SMA of %D
			push(@curK,$this->{cur_k});
            $this->{cur_d}  = intializeSMA($this->{d},@curK);
			push(@curD,$this->{cur_d});
			$this->{cur_sd} = intializeSMA($this->{sd},@curD);
        }
        elsif( @{ $this->{val} } >= $this->{days} - $this->{sd} + 1 ) {
            $this->{cur_k}  = ($this->{val}->[-1] - $this->{val_min}->[-1]) / ($this->{val_max_minus_min}->[-1]) * 100;
            $this->{cur_d}  = $this->{val_d}->[-1];
            $this->{cur_sd} = undef;
        }
        elsif( @{ $this->{val} } >= $this->{days} - $this->{sd} - $this->{d} + 2 ) {
            $this->{cur_k}  = ($this->{val}->[-1] - $this->{val_min}->[-1]) / ($this->{val_max_minus_min}->[-1]) * 100;
            $this->{cur_d}  = undef;
            $this->{cur_sd} = undef;
        }
        else {
            $this->{cur_k}  = undef;
            $this->{cur_d}  = undef;
            $this->{cur_sd} = undef;
        }
    }
    else {
        $this->{cur_k}  = undef;
        $this->{cur_d}  = undef;
        $this->{cur_sd} = undef;
    }
}

sub picklast {
    my $n = int(shift);
    return splice @_,-$n;
}

sub intializeSMA {
	my ($days,@values) = @_;
	my $sma = new Math::Business::SMA;
	$sma->set_days( $days );
	$sma->insert($_) for @values;
	my $ma = HandleResults($sma);
	return $ma;
}

sub HandleResults {
my ($sma) = @_;
my ($q,$value);
if( defined(my $q = $sma->query) ) {
      $value = $q;  

  } else {
      $value =  1;
  }
  return $value;
}

__END__

=head1 NAME

Math::Business::Stochastic - Perl extension for calculate stochastic oscillator

=head1 SYNOPSIS

  use Math::Business::Stochastic;

  my $stoc = new Math::Business::Stochastic;

  my ($k, $d, $sd) = (5, 3, 3);

  set_days $stoc $k, $d, $sd;

  my @high_values = qw(
      3 5 5 6 6 5 7 5 8 5 7
      8 6 8 6 8 7 8 8 9 8 9
  );
  my @low_values = qw(
      2 4 3 5 3 5 3 4 5 3 4
      4 5 6 6 6 6 6 7 7 6 7
  );
  my @close_values = qw(
      3 4 4 5 6 5 6 5 5 5 5
      6 6 6 6 7 7 7 8 8 8 8
  );

  for(my $i=0 ; $iinsert( $high_values[$i], $low_values[$i], $close_values[$i] );

      if( defined $stoc->query_k ) {
          print "Stochastic k:  ", $stoc->query_k,  "\n";
      }
      else {
          print "Stochastic k:  n/a\n";
      }
      if( defined $stoc->query_d ) {
          print "Stochastic d:  ", $stoc->query_d,  "\n";
      }
      else {
          print "Stochastic d:  n/a\n";
      }
      if( defined $stoc->query_sd ) {
          print "Stochastic sd: ", $stoc->query_sd,  "\n";
      }
      else {
          print "Stochastic sd: n/a\n";
      }
  }

  # you may use this to kick start
  $stoc->start_with( [@high_values], [@low_values], [@close_values] );

=head1 SEE ALSO

perl(1), Math::Business::MACD(3).

=head1 THANKS

Jettero Heller 

=head1 COPYRIGHT AND LICENSE

Copyright (C) 2008 by NAGAYASU Yukinobu 

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.8.7 or,
at your option, any later version of Perl 5 you may have available.

=cut

Greg Perl

Yahoo Stock Screening Using Perl

June 19th, 2008

So you need to screen stocks? And you want to automate the process?
Well I have not found much out there in the “Free” world that lets you do this. Yahoo seems to be about the best screening tool especially for Trend Traders as it allows you to filter on BETA. The only thing I don’t like is it doesn’t screen for Average Daily Volume. However with the script below, you could get a list of high beta stocks and then loop through them to return AVG Daily Volume.

This script parses the Yahoo Screen page by page and returns the values sorted by BETA (descending order). If you make it better, please let me know and I can post it.

In my case I only cared about the price range and a BETA > 2. However if you create your own screen you could just hack up the URL to your liking.

#!c:\\perl\\bin
use strict;
use HTML::TableExtract;
use LWP::UserAgent::Determined;

our %STOCKBETAS = ();

###These could be changed into arguments....Set them to what you wish.
my ($minprice,$maxprice,$minbeta) = (10,50,2);

####THE SCRIPT###############
my $url = 'http://screen.yahoo.com/b?pr=' . "$minprice/" . "$maxprice" . "&beta=$minbeta" . "/&s=nm&vw=1&db=stocks&b=1";
my $capture = GetStockScreen($url);
my $count = extractCount($capture); ###gets the count of symbols from the first page of the screen (i.e showing 20 of 341 symbols);
GetSymbols($capture);

for (my $x = 20;$x <= $count; $x+=20 ) { ###Loop through pages to get stock symbols
	my $b = $x + 1;
	my $MYURL = 'http://screen.yahoo.com/b?pr=' . "$minprice/" . "$maxprice" . "&beta=$minbeta" . "/&s=nm&vw=1&db=stocks&b=" . "$b";
	$capture = GetStockScreen($MYURL);
	GetSymbols($capture);
}

###Print the Results by Beta Decending.
open(SCREEN, ">YahooStockScreen.txt");
foreach my $key (sort hashValueDescendingNum  (keys(%STOCKBETAS))) {
   print SCREEN "$key\|$STOCKBETAS{$key}\n";
}
close (SCREEN);
#END OF THE SCRIPT########################################

###Retrieve the count from the first page of the yahoo screen.
sub extractCount {
my ($capture) = @_;
my $count = 1;
my $te = HTML::TableExtract->new( depth => 0, count => 0 );
$te->parse($capture);
my $table = $te->first_table_found;
	foreach my $ts ($te->tables)
	{
		#print "Table found at ", join(',', $ts->coords), ":\n";
		foreach my $row ($ts->rows) {
			    if ($count eq 1) {
					$a = substr($row->[0],index($row->[0],"of")+3, length($row->[0]));
					$count = substr($a,0,index($a,")"));
				}
				}

	}
	return $count;
}

####Retrieve the list of symbols and Betas from yahoo page
sub GetSymbols {
my ($capture) = @_;
my @symbols = ();
my $te = HTML::TableExtract->new( depth => 1, count => 0 );
$te->parse($capture);
my $table = $te->first_table_found;
	foreach my $ts ($te->tables)
	{
		#print "Table found at ", join(',', $ts->coords), ":\n";
		foreach my $row ($ts->rows) {
				my $a = $row->[0];
				my $b = $row->[3];
				$STOCKBETAS{$a}=$b;
				print "$a,$b\n";
				}
	}
	return @symbols;
}

###go to login page and login.
sub GetStockScreen {
	my ($u) = @_;
	my $response = getURL($u);
	$capture = $response ->content;	

	return $capture;
}

###Download the URL it is passed. Uses LWP::UserAgent::Determined so its on a retry interval
sub getURL {
	my ($url) = @_;
	print "GETTING: $url\n";
	my $browser = LWP::UserAgent::Determined->new;
	$browser->timing( "10,15,20,30,30,30,30,30" ); ###get url on 10,15,20,30 second retry  intervals.
	my $response   = $browser->get($url);
	return $response;
}

####Sort the output in decending order.
sub hashValueDescendingNum  {
   $STOCKBETAS{$b} <=> $STOCKBETAS{$a};
}

Greg Finance, Perl

Using WordPress::XMLRPC Perl Module for posting to a Wordpress Blog

June 12th, 2008

The CPAN documentation for this is not entirely clear. Although you can look at the source code and try to back your way out to see what the module is expecting, this should make your life easier.

Here is a quick example of how to post to a blog. The most difficult thing was adding the category. It wants an array, but you can just pass it @categories. It has to be in brackets “[ ]“.


use WordPress::XMLRPC;

my $o = WordPress::XMLRPC->new({
username => ‘your username’,
password => ‘your password’,
proxy => ‘http://www.site.com/xmlrpc.php’,
});

$content_hashref->{title} = ‘The Title for your post’;
$content_hashref->{categories} = ['Category1','Category2'];
$content_hashref->{description} = “Here you put the content of your post”;
$o->newPost($content_hashref, 1); ### 1 is for publish. I believe its 1 by default and 0 for unpublished

You can find the module on CPAN

Greg Perl ,

Login To Google Using Perl

June 6th, 2008

In this case I was particularly interested in logging in, so that I could programmatically download my Financial Portfolios, to see information the way I’d like it. I though about building my own database, but why? Google does a fantastic job of doing this for me, and as long as I can grab it with Perl (The greatest language on Earth). Then what is the point of me managing my own MySQL tables, worring about its uptime etc.

So feel free to alter the code, but this will get you into Google. You will most likely just need to alter the url lines and put in your username and password info.

Here is the code:

You will need the following Modules

IO-Socket-SSL

WWW-MECHANZE

and HTTP::Cookies;

http://gregjessup.com/snippets/google_login.txt

#!c:\\perl\\bin
use strict;
use WWW::Mechanize;
use HTTP::Cookies;
###go to login page and login.
my $url = “https://www.google.com/accounts/ServiceLogin?hl=en&service=finance&nui=1&continue=http%3A%2F%2Ffinance.google.com%2Ffinance”;
my $username = “yourname@gmail.com”;
my $password = “yourpassword”;
my $mech = WWW::Mechanize->new();
$mech->cookie_jar(HTTP::Cookies->new());
$mech->get($url);
$mech->form_number(1);
$mech->field(Email => $username);
$mech->field(Passwd => $password);
$mech->click();
#Go to the next link, now that we are logged in.
$url = ‘http://finance.google.com/finance/portfolio?action=view&pid=1&pview=pview&output=csv’;
$mech->get($url);
my $output_page = $mech->content();

#now you can do something with the content. Remember $output_page is a string :(

Greg Perl

Parse A CSV String Using Perl

June 6th, 2008

Parsing a CSV file with perl is probably one of the simpler thing’s you’ll have to do in life. You could use something as simple as

my @line = split(/,/,$somedata);

Or even simpler, you could use the Perl Package Text::CSV, which I would recommend because if you are like me you parse financial data alot! So When a company like Goldman Sachs Group, Inc. shows up…the “,” before the Inc can throw you off. However Text::CSV is smart enough to know that if the value is in quotes….not to split it.

Well the problem I ran into the other day was I was trying to login to Google Finance using perl, which was also surprisingly easy to do. I wanted just the names in my portfolio so I could use them to query other data not available the way I’d like to see it in Google. (To be more specific, I wanted a report that showed all my Dividend paying stocks. Their EX-Dates and Pay dates all in one page. )

Well Google has been nice enough to allow me to download my Portfolio in CSV format. Nice! The one problem is that when you go to the link with WWW::Mechanize, the content is returned as a string. Not a line by line file, like I am used to.

So the very, very simple solution was to split the string on New lines and store them in an Array. Then loop through each line, and use Text::CSV to give me the fields. Since I only wanted the symbol field, I only print it.

Here is the code snippet:

our $csv = Text::CSV->new();

my @line = split(/\n/,$output_page);
foreach (@line) {
my $data = $csv->parse($_);
my @fields = $csv->fields();
print “$fields[1]\n”;
}

Greg Perl