Archive

Archive for the ‘Perl’ Category

Archiving Perl PHP RUBY Scripts to Evernote using File Auto Import and Some Perl Scripting

July 25th, 2008
Archiving Perl PHP RUBY Scripts to Evernote using File Auto Import and Some Perl Scripting
Today Evernote released a new version which added the ability to drag and drop pictures or text files directly to evernote.
For a while I’ve been thinking about putting all my perl scripts in evernote, but was not in the mood to
1) Open each file
2) Copy the text
3) Paste It to Evernote
Thats also repetiteve and I don’t do Repetitive without Perl!
Evernote Offers a nice little feature File Auto Import which can be found in the menus under
Account > Properties.
The problem is, It will not import .pl file or .cgi or .php for that matter. Only .txt and images, and maybe a few others but
definitely not .pl
So, I wrote a perl script that goes through my perl script parent directory.
NOTE:(I store all my scripts for the most part in one dir, and create sub directories under it. So It was quite easy for me.)
And then makes a copy of the file in the Evernote AutoImport directory with a nice Title and the .txt extension.
##This is the path to your Evernote Import Directory
$outdir = "C:\\temp\\EvernoteScripts";
# look in current directory
$dir = `pwd`;
chop($dir);
my @files = `find | grep ".pl" | grep -v ".swp" | grep -v ".doc"`;
 
 
foreach my $f (@files) {
	chop($f);
	$filenameOnly = `basename $f`;
	$directoryOnly = `dirname $f`;
	$newfile = substr($filenameOnly,0,index($filenameOnly,"."));
	print "$filenameOnly\n";
	$text = "\# Script : $filenameOnly\n"; ### Creates a title for the script which will be seen in Evernote!
	$text = $text . `cat $f`;
	$nf = "$outdir\\$newfile\.txt";
 
	open FILE, ">", $nf or die $!;
	print FILE "$text";	
	close FILE;
	sleep 2; ###Evernote basically exploded when it tried to read from the directory without a slight delay.	
}

Greg Perl

Monitor GMAIL Account Activity Using Perl

July 9th, 2008
As recently noted over at my favorite website lifehacker.com…Google has recently added a nice feature to GMAIL that allows you to see your account activity.
If you scroll to the bottom of you Inbox, you will see something that looks like this.
This really got me thinking. This is a great feature, however its lacking in a few areas.

1) I’d like to know when suspicious activity is occuring. Although the likelyhood is low (that someone is trying to hack in…at least I  think), I’d still like to have alerts that tell me there is an issue.

2)  I’d also like tracking details in case there is an issue, I can try to figure out who it is,and where they are coming from.  Actually, when I first looked at this, I noticed someone or somethign was hitting me from 67.228.182.163. Using my technical efficentcies…I was able to track it back to Xoopit. Another service I found over at lifehacker.com.

The process was basically taking the IP in the activity window and doing a whois lookup. The problem is that the whois, generally returns the ISP. However, luckily there was an RWHOIS available. That pointed me to Xoopit!
Well, from here I decided to write a script that will run every 30 mins, Alert me if something looks suspicious.

It does the following.
1) Logs into Gmail.
2) Pulls up Account Activity Page
3) Parses Page.
4) If IP is not in the whitelist…
a. does a Whois Loopup on the IP
b. generates

5) If the count of suspicious IP’s is > 0. Sends and email using Gmail as SMTP server to whomever cares to know.
HERE IS THE CODE:
Use it as you wish. If you have issues or like it, please leave comments.
#!c:\\perl\\bin
#use strict;
use WWW::Mechanize;
use HTTP::Cookies;
use HTML::TableExtract;
use Net::Whois::IP qw(whoisip_query);
use chilkat;
our $count = 0;
my %Whitelist = 
		( 	
			'67.228.182.163' => 'Xoopit',			
			'ip.ip.ip.ip' => 'Work'			
		);
 
my $capture = GetGoogleActivity();	
my $HTML = ParseActivity($capture);
 
print "Count = $count\n";
if ($count > 0) {
	DoEmail("GMAIL: Possible Suspicious Activity", $HTML);
}
 
 
 
sub GetGoogleActivity {
###go to login page and login.
	my $url = "https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1k96igf4806cy&ltmpl=default&ltmplcache=2&hl=en";
	my $username = "someone@gmail.com";
	my $password = "password";
	my $mech = WWW::Mechanize->new();
	$mech->cookie_jar(HTTP::Cookies->new());
	$mech->get($url);
	$mech->form_number(1);
	$mech->field(Email => $username);
	$mech->field(Passwd => $password);
	$mech->click();
	#Go to the next link, now that we are logged in.
	$url = 'https://mail.google.com/mail/?ui=2&ik=46e3aa5be5&view=ac';
	$mech->get($url);
	my $output_page = $mech->content();
	return $output_page;
}
 
sub ParseActivity {
my ($capture) = @_;
 
$te = HTML::TableExtract->new( depth => 3, count => 1);
$te->parse($capture);
$table = $te->first_table_found;
foreach $ts ($te->tables) {
     foreach $row ($ts->rows) {
		#for my $r (@$row) {
			$type = @$row->[0];
			$ip = @$row->[1];
			$when = @$row->[2];
			$type = RemoveWhiteSpace($type);
			chop($ip);
			$ip = RemoveWhiteSpace($ip);
 
			$when = RemoveWhiteSpace($when);	
			print "Whitelisted IP [$ip] = $Whitelist{$ip}\n";
			if ($ip !=~ "?") { 
					if (! $Whitelist{$ip}) {
						#if (($ip !=~ "?") && (! $Whitelist{$ip})) {
						$HTML = $HTML . "<table broder=\"1\">\n<th colspan=\"3\" align=\"left\">Connection From $ip</th>\n<tr>\n<td>$type</td><td>$when</td><td>$ip</td>\n</tr>\n";
						#print "$type\t$when\t$ip\n\n";
						$HTML = Whois($ip,$HTML);
						$HTML = $HTML . "</table>\n";
					}
			}		
				#}
 
 
			#$html = "$html" . "$town \t $d<br>\n";
			#$count++;
		}
	}
	return $HTML;	
}
 
sub Whois {
	$count++;
	my ($IP,$html) = @_;	
	print "Looking up $IP\n";
	my $response = whoisip_query($IP); 
	foreach (sort keys(%{$response}) ) { 
		$html = $html . "<tr><td colspan=\"2\">$_</td><td>$response->{$_}</td></tr>\n";		 
	}
	return $html;
}
 
sub RemoveWhiteSpace {
	my ($val) = @_;
	for ($val) {
		s/\*//; #Also removes *
		s/^\s+//;
		s/\s+$//;		
	}
return $val;
}
 
sub DoEmail {
my ($title,$description) = @_;
# file: GMail.pl
print "$title\n $description\n";
# Perl script to send email using GMail as the SMTP server.	
 
 
$mailman = new chilkat::CkMailMan();
$mailman->UnlockComponent('anything for 30-day trial');
 
# Set your SMTP server's hostname
$mailman->put_SmtpHost('smtp.gmail.com');
 
# GMail requires a login/password to send mail.
# Strings containing a '@' should always be in single quotes.
$mailman->put_SmtpUsername('someone@gmail.com');
$mailman->put_SmtpPassword("password");
 
# The default SMTP port is 25.  When using it, GMail requires STARTTLS.
$mailman->put_StartTLS(true);
 
# Alternatively, you may comment-out the STARTTLS line and instead use SSL
# on port 465 by commenting-in these 2 lines:
# $mailman->put_SmtpPort(465);
# $mailman->put_SmtpSsl(true);
 
# If you are connected to a network that blocks outbound port 25 connections,
# use GMail's alternative port 587.  You'll need STARTTLS, so uncomment the 
# STARTTLS line and make sure the two lines for SMTP SSL are commented out.
$mailman->put_SmtpPort(587);
 
$mht = new chilkat::CkMht();
$mht->UnlockComponent('anything for 30-day trial');
 
open(INFO, ">GmailActivity.html");
$body = "<html>
<head>
<html>$description
</html>";
print INFO "$body";
close INFO;
 
# Instantiate a new email object.
$email = new chilkat::CkEmail();
$email = $mht->GetEmail('GmailActivity.html');
$email->put_Subject($title);
#$email->put_Body($body);
$email->put_From('Greg The Great! <someone@gmail.com>');
 
# Add some recipients
$email->AddTo('Email','someone@gmail.com');
 
 
 
$success = $mailman->SendEmail($email);
if (! $success)
    {
	$mailman->SaveLastError('lastError.txt');	
    }
else 
{
	# The log will contain information allowing you to verify
	# what actually happened.
	# If you see an error "Error authenticating server credentials!" this is 
	# normal and OK.
	$mailman->SaveLastError('lastInfo.txt');	
}
 
}

Greg Perl

Bypass Proxy with a Perl and Firefox Hack

June 25th, 2008

So being a developer, I rely on both my smarts and the smarts of others to get the job done.  One of the greatest ways to come up with a solution is using google to search to see how others have done similar things with code, and then crafting your own solution. The problem is many companies block blogspot, wordpress and other blogging urls thus making it impossible for you to access some of the very useful blogs out there. Luckily I run my blog off my own URL so its probably not blocked.

Well after enough frustration, I decided it was time to fix this problem. I spend 10-12hrs a day at the office, I don’t want to wait to get home to figure do my research.

I found a cgi script that acts as an http or ftp proxy from www.jmarshall.com/tools/cgiproxy/

Because I have a couple of websites up and running I had a readily available cgi-bin directory. I simply dropped the script on my server and WA-LA! I was given a form field which acted as an address bar. If I copy and pasted a blog link into this field, I was taken to the site! AWESOME!

I must mention one more thing. It actually wasn’t quite that simple. Websense was very smart at my company, and probably is at yours as well. When I first ran it, it said “Proxy Avoidence Detected!!”.

Ouch!!!!!!

Well, I obviously was not completely out of luck or I would not be writing this post…now would I?

The very simple solution which popped in my head within about 1 second of getting this error was……

You got it! Run in on a secure channel. HTTPS. This will require a certificate, but this I also already had with my ISP. So all I had to do was forward to HTTPS!

GREAT!!! I’m finally up an running!

After about a week of copying the links I found, Then visiting my https site, then pasting to the input box….I realized this was just too many steps. Ideally I’d be able to right click a link and say “OPEN WITH PROXY HACK” (and that is my next project). However, this was going to require me to learn to write a firefox extension (which i have not done yet).

So I opted for the very simple solution of adding it to my search engine drop down list on the upper right hand corner of Firefox.

Now I simply copy the link and paste it up in my search box and it forwards me to the site.

Here is the code for the FireFox Search Engine….You will just need to..
1) Edit the links.
2) Save the file as ProxyHack.xml (or whatever you’d like it to be.xml)
3) Copy the file to Path to Firefox\searchplugins
4) Restart Firefox!
5) Send me a thank you :)

Greg Perl

Perl Stochastic Math::Business::Stochastic

June 20th, 2008

I recently found a bug in the Perl Stochastic calculator which I have amended for myself and am now sharing with the world.

%D should be an xDay moving average of %K and SD should be the Moving average of %D. I have added 2 subroutines that make use of Math::Business::SMA.

I keep an array of %K values and pass it to my intializeSMA subroutine. This creates a SMA for %K based on $this->{d}. I do the same for SD. I keep an array of %D and then

keep calculate a SMA based on SD $this->{sd}.

Here is the updated Perl Module:

package Math::Business::Stochastic;

use strict;
use warnings;
use diagnostics;

our $VERSION = '0.03';
our ($kkk,@curK,@curD);
use Carp;
use List::Util qw(max min sum);

1;

sub new {
    bless {
        val  => [],
        days => 0,
    };

}

sub set_days {
    my $this = shift;
    my ($k,$d,$sd) = @_;

    croak "k must be a positive non-zero integers"  if int($k) <= 0;
    croak "d must be a positive non-zero integers"  if int($d) <= 0;
    croak "sd must be a positive non-zero integers" if int($sd) <= 0;

    $this->{k} = int($k);
    $this->{d} = int($d);
    $this->{sd} = int($sd);
    $this->{days} = int($k) + int($d) + int($sd) - 2;
	print "DAYS = " . $this->{days} . "\n";
}

sub query_k  { my $this = shift; return $this->{cur_k}; }
sub query_d  { my $this = shift; return $this->{cur_d}; }
sub query_sd { my $this = shift; return $this->{cur_sd}; }

sub insert {
    my $this = shift;
    my ($high,$low,$close) = @_;

    croak "You must set the number of days before you try to insert" if not $this->{days};
    croak "You must specify the high,low,close values" unless defined $close;
    croak "High value must be higher than low value" unless $high >= $low;
    croak "Low value must be lower than close value" unless $low <= $close;
    croak "High value must be higher than close value" unless $high >= $close;

    push @{ $this->{val_high} }, $high;
    push @{ $this->{val_low} }, $low;
    push @{ $this->{val} }, $close;

    $this->recalc;
}

sub start_with {
    my $this        = shift;
       $this->{val_high} = shift;
       $this->{val_low}  = shift;
       $this->{val}      = shift;

    croak "bad arg to start_with" unless ref($this->{val_high}) eq "ARRAY";
    croak "bad arg to start_with" unless ref($this->{val_low}) eq "ARRAY";
    croak "bad arg to start_with" unless ref($this->{val}) eq "ARRAY";
    croak "bad arg to start_with" unless @{$this->{val_high}} == @{$this->{val}};
    croak "bad arg to start_with" unless @{$this->{val_low}} == @{$this->{val}};

    $this->recalc;
}

sub recalc {
    my $this = shift;
    shift @{ $this->{val_high} } while @{ $this->{val_high} } > $this->{days};
    shift @{ $this->{val_low} } while @{ $this->{val_low} } > $this->{days};
    shift @{ $this->{val} } while @{ $this->{val} } > $this->{days};

    if( $this->{k} <= @{ $this->{val} }  ) {

        push @{ $this->{val_max} }, max( picklast($this->{k},@{$this->{val_high}}) );
        push @{ $this->{val_min} }, min( picklast($this->{k},@{$this->{val_low}}) );
        push @{ $this->{val_close_minus_min} }, $this->{val}->[-1] - $this->{val_min}->[-1];
        push @{ $this->{val_max_minus_min} }, $this->{val_max}->[-1] - $this->{val_min}->[-1];
        shift @{ $this->{val_max} } while @{ $this->{val_max} } > $this->{k};
        shift @{ $this->{val_min} } while @{ $this->{val_min} } > $this->{k};
        shift @{ $this->{val_close_minus_min} } while @{ $this->{val_close_minus_min} } > $this->{k};
        shift @{ $this->{val_max_minus_min} } while @{ $this->{val_max_minus_min} } > $this->{k};
    }
    if( $this->{k}+$this->{d}-1 <= @{ $this->{val} } ) {

	   push @{ $this->{val_d} }, sum(picklast($this->{d},@{$this->{val_close_minus_min}})) / sum(picklast($this->{d},@{$this->{val_max_minus_min}})) * 100;
        shift @{ $this->{val_d} } while @{ $this->{val_d} } > $this->{sd};
    }

    if( not defined $this->{val_max_minus_min}->[-1] or $this->{val_max_minus_min}->[-1] > 0 ) {

		if( @{ $this->{val} } == $this->{days} ) {
            $this->{cur_k}  = ($this->{val}->[-1] - $this->{val_min}->[-1]) / ($this->{val_max_minus_min}->[-1]) * 100;
			###This was added 6/20/2008 Keeps an array of %K and calculates %D as a SMA of %K then calculates Slow Stochastics using A SMA of %D
			push(@curK,$this->{cur_k});
            $this->{cur_d}  = intializeSMA($this->{d},@curK);
			push(@curD,$this->{cur_d});
			$this->{cur_sd} = intializeSMA($this->{sd},@curD);
        }
        elsif( @{ $this->{val} } >= $this->{days} - $this->{sd} + 1 ) {
            $this->{cur_k}  = ($this->{val}->[-1] - $this->{val_min}->[-1]) / ($this->{val_max_minus_min}->[-1]) * 100;
            $this->{cur_d}  = $this->{val_d}->[-1];
            $this->{cur_sd} = undef;
        }
        elsif( @{ $this->{val} } >= $this->{days} - $this->{sd} - $this->{d} + 2 ) {
            $this->{cur_k}  = ($this->{val}->[-1] - $this->{val_min}->[-1]) / ($this->{val_max_minus_min}->[-1]) * 100;
            $this->{cur_d}  = undef;
            $this->{cur_sd} = undef;
        }
        else {
            $this->{cur_k}  = undef;
            $this->{cur_d}  = undef;
            $this->{cur_sd} = undef;
        }
    }
    else {
        $this->{cur_k}  = undef;
        $this->{cur_d}  = undef;
        $this->{cur_sd} = undef;
    }
}

sub picklast {
    my $n = int(shift);
    return splice @_,-$n;
}

sub intializeSMA {
	my ($days,@values) = @_;
	my $sma = new Math::Business::SMA;
	$sma->set_days( $days );
	$sma->insert($_) for @values;
	my $ma = HandleResults($sma);
	return $ma;
}

sub HandleResults {
my ($sma) = @_;
my ($q,$value);
if( defined(my $q = $sma->query) ) {
      $value = $q;  

  } else {
      $value =  1;
  }
  return $value;
}

__END__

=head1 NAME

Math::Business::Stochastic - Perl extension for calculate stochastic oscillator

=head1 SYNOPSIS

  use Math::Business::Stochastic;

  my $stoc = new Math::Business::Stochastic;

  my ($k, $d, $sd) = (5, 3, 3);

  set_days $stoc $k, $d, $sd;

  my @high_values = qw(
      3 5 5 6 6 5 7 5 8 5 7
      8 6 8 6 8 7 8 8 9 8 9
  );
  my @low_values = qw(
      2 4 3 5 3 5 3 4 5 3 4
      4 5 6 6 6 6 6 7 7 6 7
  );
  my @close_values = qw(
      3 4 4 5 6 5 6 5 5 5 5
      6 6 6 6 7 7 7 8 8 8 8
  );

  for(my $i=0 ; $iinsert( $high_values[$i], $low_values[$i], $close_values[$i] );

      if( defined $stoc->query_k ) {
          print "Stochastic k:  ", $stoc->query_k,  "\n";
      }
      else {
          print "Stochastic k:  n/a\n";
      }
      if( defined $stoc->query_d ) {
          print "Stochastic d:  ", $stoc->query_d,  "\n";
      }
      else {
          print "Stochastic d:  n/a\n";
      }
      if( defined $stoc->query_sd ) {
          print "Stochastic sd: ", $stoc->query_sd,  "\n";
      }
      else {
          print "Stochastic sd: n/a\n";
      }
  }

  # you may use this to kick start
  $stoc->start_with( [@high_values], [@low_values], [@close_values] );

=head1 SEE ALSO

perl(1), Math::Business::MACD(3).

=head1 THANKS

Jettero Heller 

=head1 COPYRIGHT AND LICENSE

Copyright (C) 2008 by NAGAYASU Yukinobu 

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.8.7 or,
at your option, any later version of Perl 5 you may have available.

=cut

Greg Perl

Yahoo Stock Screening Using Perl

June 19th, 2008

So you need to screen stocks? And you want to automate the process?
Well I have not found much out there in the “Free” world that lets you do this. Yahoo seems to be about the best screening tool especially for Trend Traders as it allows you to filter on BETA. The only thing I don’t like is it doesn’t screen for Average Daily Volume. However with the script below, you could get a list of high beta stocks and then loop through them to return AVG Daily Volume.

This script parses the Yahoo Screen page by page and returns the values sorted by BETA (descending order). If you make it better, please let me know and I can post it.

In my case I only cared about the price range and a BETA > 2. However if you create your own screen you could just hack up the URL to your liking.

#!c:\\perl\\bin
use strict;
use HTML::TableExtract;
use LWP::UserAgent::Determined;

our %STOCKBETAS = ();

###These could be changed into arguments....Set them to what you wish.
my ($minprice,$maxprice,$minbeta) = (10,50,2);

####THE SCRIPT###############
my $url = 'http://screen.yahoo.com/b?pr=' . "$minprice/" . "$maxprice" . "&beta=$minbeta" . "/&s=nm&vw=1&db=stocks&b=1";
my $capture = GetStockScreen($url);
my $count = extractCount($capture); ###gets the count of symbols from the first page of the screen (i.e showing 20 of 341 symbols);
GetSymbols($capture);

for (my $x = 20;$x <= $count; $x+=20 ) { ###Loop through pages to get stock symbols
	my $b = $x + 1;
	my $MYURL = 'http://screen.yahoo.com/b?pr=' . "$minprice/" . "$maxprice" . "&beta=$minbeta" . "/&s=nm&vw=1&db=stocks&b=" . "$b";
	$capture = GetStockScreen($MYURL);
	GetSymbols($capture);
}

###Print the Results by Beta Decending.
open(SCREEN, ">YahooStockScreen.txt");
foreach my $key (sort hashValueDescendingNum  (keys(%STOCKBETAS))) {
   print SCREEN "$key\|$STOCKBETAS{$key}\n";
}
close (SCREEN);
#END OF THE SCRIPT########################################

###Retrieve the count from the first page of the yahoo screen.
sub extractCount {
my ($capture) = @_;
my $count = 1;
my $te = HTML::TableExtract->new( depth => 0, count => 0 );
$te->parse($capture);
my $table = $te->first_table_found;
	foreach my $ts ($te->tables)
	{
		#print "Table found at ", join(',', $ts->coords), ":\n";
		foreach my $row ($ts->rows) {
			    if ($count eq 1) {
					$a = substr($row->[0],index($row->[0],"of")+3, length($row->[0]));
					$count = substr($a,0,index($a,")"));
				}
				}

	}
	return $count;
}

####Retrieve the list of symbols and Betas from yahoo page
sub GetSymbols {
my ($capture) = @_;
my @symbols = ();
my $te = HTML::TableExtract->new( depth => 1, count => 0 );
$te->parse($capture);
my $table = $te->first_table_found;
	foreach my $ts ($te->tables)
	{
		#print "Table found at ", join(',', $ts->coords), ":\n";
		foreach my $row ($ts->rows) {
				my $a = $row->[0];
				my $b = $row->[3];
				$STOCKBETAS{$a}=$b;
				print "$a,$b\n";
				}
	}
	return @symbols;
}

###go to login page and login.
sub GetStockScreen {
	my ($u) = @_;
	my $response = getURL($u);
	$capture = $response ->content;	

	return $capture;
}

###Download the URL it is passed. Uses LWP::UserAgent::Determined so its on a retry interval
sub getURL {
	my ($url) = @_;
	print "GETTING: $url\n";
	my $browser = LWP::UserAgent::Determined->new;
	$browser->timing( "10,15,20,30,30,30,30,30" ); ###get url on 10,15,20,30 second retry  intervals.
	my $response   = $browser->get($url);
	return $response;
}

####Sort the output in decending order.
sub hashValueDescendingNum  {
   $STOCKBETAS{$b} <=> $STOCKBETAS{$a};
}

Greg Finance, Perl

Using WordPress::XMLRPC Perl Module for posting to a Wordpress Blog

June 12th, 2008

The CPAN documentation for this is not entirely clear. Although you can look at the source code and try to back your way out to see what the module is expecting, this should make your life easier.

Here is a quick example of how to post to a blog. The most difficult thing was adding the category. It wants an array, but you can just pass it @categories. It has to be in brackets “[ ]“.


use WordPress::XMLRPC;

my $o = WordPress::XMLRPC->new({
username => ‘your username’,
password => ‘your password’,
proxy => ‘http://www.site.com/xmlrpc.php’,
});

$content_hashref->{title} = ‘The Title for your post’;
$content_hashref->{categories} = ['Category1','Category2'];
$content_hashref->{description} = “Here you put the content of your post”;
$o->newPost($content_hashref, 1); ### 1 is for publish. I believe its 1 by default and 0 for unpublished

You can find the module on CPAN

Greg Perl ,

Login To Google Using Perl

June 6th, 2008

In this case I was particularly interested in logging in, so that I could programmatically download my Financial Portfolios, to see information the way I’d like it. I though about building my own database, but why? Google does a fantastic job of doing this for me, and as long as I can grab it with Perl (The greatest language on Earth). Then what is the point of me managing my own MySQL tables, worring about its uptime etc.

So feel free to alter the code, but this will get you into Google. You will most likely just need to alter the url lines and put in your username and password info.

Here is the code:

You will need the following Modules

IO-Socket-SSL

WWW-MECHANZE

and HTTP::Cookies;

http://gregjessup.com/snippets/google_login.txt

#!c:\\perl\\bin
use strict;
use WWW::Mechanize;
use HTTP::Cookies;
###go to login page and login.
my $url = “https://www.google.com/accounts/ServiceLogin?hl=en&service=finance&nui=1&continue=http%3A%2F%2Ffinance.google.com%2Ffinance”;
my $username = “yourname@gmail.com”;
my $password = “yourpassword”;
my $mech = WWW::Mechanize->new();
$mech->cookie_jar(HTTP::Cookies->new());
$mech->get($url);
$mech->form_number(1);
$mech->field(Email => $username);
$mech->field(Passwd => $password);
$mech->click();
#Go to the next link, now that we are logged in.
$url = ‘http://finance.google.com/finance/portfolio?action=view&pid=1&pview=pview&output=csv’;
$mech->get($url);
my $output_page = $mech->content();

#now you can do something with the content. Remember $output_page is a string :(

Greg Perl

Parse A CSV String Using Perl

June 6th, 2008

Parsing a CSV file with perl is probably one of the simpler thing’s you’ll have to do in life. You could use something as simple as

my @line = split(/,/,$somedata);

Or even simpler, you could use the Perl Package Text::CSV, which I would recommend because if you are like me you parse financial data alot! So When a company like Goldman Sachs Group, Inc. shows up…the “,” before the Inc can throw you off. However Text::CSV is smart enough to know that if the value is in quotes….not to split it.

Well the problem I ran into the other day was I was trying to login to Google Finance using perl, which was also surprisingly easy to do. I wanted just the names in my portfolio so I could use them to query other data not available the way I’d like to see it in Google. (To be more specific, I wanted a report that showed all my Dividend paying stocks. Their EX-Dates and Pay dates all in one page. )

Well Google has been nice enough to allow me to download my Portfolio in CSV format. Nice! The one problem is that when you go to the link with WWW::Mechanize, the content is returned as a string. Not a line by line file, like I am used to.

So the very, very simple solution was to split the string on New lines and store them in an Array. Then loop through each line, and use Text::CSV to give me the fields. Since I only wanted the symbol field, I only print it.

Here is the code snippet:

our $csv = Text::CSV->new();

my @line = split(/\n/,$output_page);
foreach (@line) {
my $data = $csv->parse($_);
my @fields = $csv->fields();
print “$fields[1]\n”;
}

Greg Perl

Perl HTML::TableExtract Table Finder Loop

May 23rd, 2008

This script parses a web page and prints out all of the coordinates for tables found. This should make your life easier as far as determining the location of the table you are looking to parse.

#!c:\perl\bin\perl.exe

use LWP::Simple;
use LWP::UserAgent;
use HTML::TableExtract;

use CGI;
use CGI::Carp qw(fatalsToBrowser);
my $cgi = new CGI;

my $url = “http://www.drudgereport.com/”;
my $capture = get($url);

$depth = 0;
$count = 0;

while ($depth <= 10) {
DoLoop();
$depth++;
}

sub DoLoop {
for ($i = 0; $i <= 10; $i++)
{
$te = HTML::TableExtract->new( depth => “$depth”, count => $i );
$te->parse($capture);
my $table = $te->first_table_found;
foreach my $ts ($te->tables)
{
print “Table found at “, join(’,', $ts->coords), “:\n”;
foreach $row ($ts->rows) {
$x = 0;
print join(’,', @$row), “\n”;

}
}
}

}

Greg Perl

Yahoo Finance Economic Events to XML with Perl

May 23rd, 2008

This is a very simple script which gets yahoo economic events from the yahoo finance website and converts it to XML.


#!c:\perl\bin\perl.exe

use LWP::Simple;
use LWP::UserAgent;
use HTML::TableExtract;
use CGI;
use CGI::Carp qw(fatalsToBrowser);
my $cgi = new CGI;

my $url = “http://biz.yahoo.com/c/e.html”;
my $capture = get($url);
#http://www.treasurydirect.gov/xml/PendingAuctions.xml
print $cgi->header(-type => ‘application/xml’); # make browser expect xml

$te = HTML::TableExtract->new( depth => 0, count => 5 );
print “\n”;
$te->parse($capture);
my $table = $te->first_table_found;
foreach my $ts ($te->tables)
{
#print “Table found at “, join(’,', $ts->coords), “:\n”;
#print @$row;
$count = 0;
foreach $row ($ts->rows) {
$a = @$row->[0];
$b = @$row->[1];
$c = @$row->[2];
$d = @$row->[3];
$e = @$row->[4];
$f = @$row->[5];
$g = @$row->[6];
$h = @$row->[7];
$i = @$row->[8];
#print “$a,$b,$c,$d,$e,$f,$g,$h,$i\n”;
if ($count > 0) { ###skip first line
printXML();
}
$count++
#print join(’,', @$row), “\n”;
}
}

print “\n”;
sub printXML {
print “\t\n”;
print “\t\t$a\n” .
“\t\t\n” .
“\t\t$c\n” .
“\t\t$d\n” .
“\t\t$e\n” .
“\t\t$f\n” .
“\t\t$g\n” .
“\t\t

$h\n” .
“\t\t$i\n”;
print “\t\n”
}sub printXMLHeader {
print $cgi->header(-type => ‘application/xml’); # make browser expect xml
#print “\<\?xml version=\”1.0\” encoding=\”UTF-8\”\?>\n”;
}

Greg Finance, Perl