June 2009 Archives

I've enabled OpenID authentication so you can comment more easily.  Go nuts!

NOTE: I'd like to consider this a sort of RFC as well, comments welcome!

Wow, so after a nice ass kicking by my mentor Matt Trout, I hunkered down and whipped out a pretty tangible set of code for Catalyst::Script and Catalyst::ScriptRunner.

Here's the rundown.

Instead of having our scripts do a bunch of work, I've moved all the script guts out of the scripts, and created a class for each of them, and Moosificated them.  So we have Catalyst::Script::Test, Catalyst::Script::Create, Catalyst::Script::Server, Catalyst::Script::FastCGI and Catalyst::Script::Test.  These each contain the functionality that the scripts originally had, wrapped in a nice little Moose container.  Catalyst::ScriptRunner simply takes the requested class, and runs it with arguments thanks to MooseX::Getopt. I've finished with all but Catalyst::Script::Create, and I'll surely have to branch MooseX::Getopt and hack on it to get it to do what I want.  Most of my tests pass, but the ones that don't will be fixed soon.

So, how does it look?
Like this:

Old myapp_fastcgi.pl:

#!/usr/bin/perl -w

BEGIN { $ENV{CATALYST_ENGINE} ||= 'FastCGI' }

use strict;
use warnings;
use Getopt::Long;
use Pod::Usage;
use FindBin;
use lib "$FindBin::Bin/../lib";
use MyApp;

my $help = 0;
my ( $listen, $nproc, $pidfile, $manager, $detach, $keep_stderr );

GetOptions(
    'help|?'      => \$help,
    'listen|l=s'  =>; \$listen,
    'nproc|n=i'   => \$nproc,
    'pidfile|p=s' => \$pidfile,
    'manager|M=s' => \$manager,
    'daemon|d'    => \$detach,
    'keeperr|e'   => \$keep_stderr,
);

pod2usage(1) if $help;

MyApp->run( 
    $listen, 
    {   nproc   =>$nproc,
        pidfile => $pidfile, 
        manager =>$manager,
        detach  => $detach,
    keep_stderr => $keep_stderr,
    }
);

1;

New myapp_fastcgi.pl:

#!/usr/bin/env perl
use FindBin qw/$Bin/;
use Catalyst::ScriptRunner;
Catalyst::ScriptRunner->run('Catalyst','FastCGI');
1;

Much better!

The code to implement it is fairly simple as well, thanks to Moose:

package Catalyst::ScriptRunner;
use Moose;

sub run {
    my ($self, $class, $scriptclass) = @_;
    my $classtoload = "${class}::Script::$scriptclass"; 
    Class::MOP::load_class($classtoload); 
    $classtoload->new_with_options->run;
}
1;

And Catalyst::Script::FastCGI:

package Catalyst::Script::FastCGI;

BEGIN { $ENV{CATALYST_ENGINE} ||= 'FastCGI' }
use FindBin qw/$Bin/;
use lib "$Bin/../lib";
use Pod::Usage;
use Moose;
use namespace::clean -except => [ qw(meta) ];

with 'MooseX::Getopt';

has help        => ( isa => 'Bool',   is => 'ro', required => 0, default => sub { 0 } );
has listen      => ( isa => 'Int',    is => 'ro', required => 1 );
has pidfile     => ( isa => 'Str',    is => 'ro', required => 0 );
has daemon      => ( isa => 'Bool',   is => 'ro', required => 0, default => sub { 0 } );
has manager     => ( isa => 'Str',    is => 'ro', required => 0 );
has keep_stderr => ( isa => 'Bool',   is => 'ro', required => 0 );
has nproc       => ( isa => 'Int',    is => 'ro', required => 0 );
has detach      => ( isa => 'Bool',   is => 'ro', required => 0, default => sub { 0 } );
has app         => ( isa => 'Str',    is => 'ro', required => 1 );

sub run {
    my $self = shift;
    
    pod2usage() if $self->help;
    my $app = $self->app;
    Class::MOP::load_class($app);
    $app->run(
        $self->listen,
        {  
            nproc   => $self->nproc,
            pidfile => $self->pidfile,
            manager => $self->manager,
            detach  => $self->detach,
            keep_stderr => $self->keep_stderr,
        }  
    );

}

This is really cool, as it allows for SO much more flexibility, and a lot more clean code.  Users can even drop in their own scripts if they pass it to Catalyst::ScriptRunner->run!

This is a technical write up of my current status on Google Summer of Code.
I figured it would make sense to start by comparing their timeline to mine, and then comparing where I'm at to both.
Google's timeline: http://socghop.appspot.com/document/show/program/google/gsoc2009/faqs#timeline

My timline: (hopefully this is viewable to the public) http://socghop.appspot.com/student_proposal/show/google/gsoc2009/dhoss/t123871990398

Where I currently am (dump of my "done list"):
List of what I've done on Catalyst::Helper thus far:

6-03-2009
1. removed template code and hex data for images and put them into a directory
structure reflecting a Catalyst application's set up
2. Written a test to check to make sure the original C::H and new C::H code
create the exact same application, so we know nothing was broken.
3.  Implemented File::ShareDir to make use of a share/ directory in a distribution
for template files and images, etc.
4. working on a test script to test how deprecated methods are invoked, 2/3 working
5. implemented methods for new API and old deprecated methods.  1 for pulling
File::ShareDir data out of our share/ directory, and another for backwards compatibility,
pulling data out of the __DATA__ section.
6. Drafted an RFC for more comments on progression of C::H API
7. Also using added Path::Class for file reading
as of 6-9-09
################################################################################
1. backwards compatibility tests pass
################################################################################
2009-06-17
         - added File::ShareDir capabilities
         - added Path::Class::Dir and Path::Class:File for ease of F::ShareDir
           file manip
         - modified the following methods:
         --  _mk_appclass    - sends the new filename to be rendered
         --  _mk_rootclass   - ''
         --  _mk_makefile    - ''
         --  _mk_config      - ''
         --  _mk_readme      - ''
         --  _mk_changes     - ''
         --  _mk_apptest     - ''
         --  _mk_cgi         - ''
         --  _mk_fastcgi     - ''
         --  _mk_server      - ''
         --  _mk_test        - ''
         --  _mk_create      - ''
         --  _mk_compclass   - ''
         --  _mk_comptestsub - ''
         --  _mk_images      - ''
         --  _mk_images      - ''
         --  _mk_favicon     - ''
         --  render_file     - this now calls render_file_contents to DTRT
         - new methods:
         -- get_sharedir_file    - this grabs a file out of share/ dir
         -- render_sharedir_file - this renders our sharedir file properly, using render_file_contents
################################################################################
2009-06-23
tests written for:
render_file_contents
render_sharedir_file
checking to see if generated app's tests pass (t/generated_app.t)
fixed paths in Helper.pm so they're windows friendly. need to do so in tests.
moosified server script
added test to make sure server script works
created branch to Moosify helper API

From what I can see, aside from some clean up and paperwork for Google, I'm doing okay.  Most of the yak shaving is complete, and the API is shaping up quiet well.  We're Moosificating the scripts (create, server, etc) to keep the busy work in the API and out of the scripts.  It wasn't originally in the plan, but it came up as something that would be pretty beneficial.  Less brain work  == good.  Nothing much to be unexpected from here on out, unless something goes wrong.

Questions

Category: Mentor Relations

Question 1: At what point did you first make contact with your mentoring organization?
A. Before the program was announced for this year (end of February 2008).
- I've been an avid perl programmer for a while, and have actually been involved with The Perl Foundation since before I applied for GSoC

Question 2: At what point did you first make contact with your mentor?
A. Before the program was announced for this year (end of February 2008).
- I've lurked in #catalyst for a while, and Kieren and I talked a number of times.  He actually was one of the people that encouraged me to apply for GSoC

Comments (optional - free text field)

Question 3: How often do you and your mentor interact?

B. Once every few days.

Comments (optional - free text field)

Question 4: How do you communicate with your mentor?  Please check all that apply.

B. IRC/Instant Messaging
E. Project wide mailing lists such as the main developers list


Comments (optional - free text field)

Question 5: Of the different communication methods you use with your mentor, which do you use most frequently?

B. IRC/Instant Messaging

 

Comments (optional - free text field)

Question 6: Which of the following means do you use to send status updates to your mentor?

C. I post their my updates to their individual blog.

F. I send my status updates to project wide mailing list such as our main developers mailing list.

H. I have a regular 1/1 IRC/other online status meeting with my mentor.
I. My mentor holds regular "office hours" online and I drop in when I have questions.

Comments (optional - free text field)

Question 7: How often do you make status updates?

C. 2 - 3 times per week

Comments (optional - free text field)

Question 8: How would you rate the amount and quality of interaction with your mentor?
A. I would prefer more interaction with my mentor, but the quality of our interaction is high.


Comments (optional - free text field)

Question 9: How would you describe your interaction with your overall project community?
A. Very active (e.g. I send code reviews to development mailing lists, am active and ask/answer questions in our IRC channel or project forums)


Comments (optional - free text field)

Question 10: If you cannot reach your mentor, how do you go about getting help when you need it?  Please check all that apply.
A. I ask my co-mentor or my assigned backup mentor.
B. I ask my organization administrator.
C. I ask questions on our project's developer or other main mailing list..
E. I ask questions in IRC or on the project forums.
F. I hope I can figure it out myself before my mentor resurfaces.

Comments (optional - free text field):

Question 11: If your mentor goes on vacation or life otherwise happens, what is your organization's plan for helping you?
A. I have a co-mentor/backup mentor to work with.
B. My organization administrator will step in as my mentor.
C. I don't know what the plan is, but I can always reach my organization administrator if something comes up.
D. I don't know what the plan is and I wouldn't know whom to ask.


Comments (optional - free text field)

Category: Your Project
Question 1: Do you feel that you are on track to complete your project?

C. I am on schedule.

Comments (optional - free text field):


Question 2: If you feel that you are behind schedule, have you worked with your mentor to reset milestones or otherwise alter project goals?
A. Not applicable, I am on track.

Comments (optional - free text field)

Question 3: How would you describe the quality of your work thus far?
D. Does not meet my personal expecations and I am working on improving

Comments (optional - free text field)

Question 4: How would you describe the effort you have put in thus far?
A. Meets my personal expectations

Comments (optional - free text field)

Question 5: How responsive was your mentor to any questions or other requests for communication from you?
A. Mentor responded quickly with the information I requested.

Comments (optional - free text field)

Question 6: Would you want to work with this mentor again?
A. Yes


Comments (optional - free text field)

Question 7: Would you want to work with this mentor on a daily basis, e.g. sitting next to you in an office?
A. Yes


Comments (optional - free text field)

Category: Program Questions

Question 1: We made several changes to the program timings last year, including announcing the program earlier in the year and adding the community bonding phase to the timeline.  Do you feel the schedule changes have been helpful?
A. Both announcing the program earlier and adding the community bonding period have been helpful.

Comments (REQUIRED)

- I was happy to see I had some good time to get prepared.  I procrastinate usually, so it really helped me to be made aware well in advance.

Question 2: What areas were the most difficult for you in preparing for/working on Summer of Code?  Please select all that apply.
A. Writing your application
C. Choosing the best idea to apply for
E. Getting up to speed with my organization's code base
H. Keeping myself motivated during school overlaps with the program timeline

M. Time zone differences

Comments (REQUIRED)

- Perl is an amazing organization, I just wish we could all be in the same time zone :-)

Question 3: What advice would you give to future would-be Summer of Code mentoring organizations? (REQUIRED)

- Be organized and ready to go.  Know what you want done, and have more people involved than just the designated mentors.  Information should be wealthy and readily available for the student.

Question 4: What advice would you give to future would-be Summer of Code students who - would like to work with your organization? (REQUIRED)

- Be ready to work hard, and have fun.  Ask a lot of questions, and make sure you understand what you're doing.  GET A HIVEMINDER (or equivelant) ACCOUNT.  You *need* a todo list.  It will save you.

Category: Overall Sanity Check
Question 1: Do you feel that Summer of Code is going well enough for you?
A. Yes

Comments (optional - free text field)

Google Summer of Code is an excellent and wonderful thing.  I'm so happy I've been chosen and have been able to embark on this adventure.

It's been an extremely long day (though condensed and compacted) of hacking on Catalyst::Helper.

I mostly worked on a test to check if the file contents of a file created with the now deprecated API were the same as the original helper API, so we could make sure we didn't break anything. 2/3 tests pass, but this last one is being absolutely heinous.

Here's some example code:

sub render_file {
    my ( $self, $file, $path, $vars ) = @_;
    my $template = $self->get_file( ( caller(0) )[0], $file );
    $self->render_file_contents($self, $template, $path, $vars);
}

sub render_sharedir_file {
    my ( $self, $file, $path, $vars ) = @_;
    my $template = $self->get_sharedir_file( $file );
    $self->render_file_contents($self, $template, $path, $vars);
}

sub render_file_contents {
    my ( $self, $template, $path, $vars ) = @_;
    $vars ||= {};
    my $t = Template->new;
    return 0 unless $template;
    my $output;
    $t->process( \$template, { %{$self}, %$vars }, \$output )
      || Catalyst::Exception->throw(
        message => qq/Couldn't process "$template", / . $t->error() );
    $self->mk_file( $path, $output );
}

This is the render_file method broken up into the new API and the backwards compatible API.  I'm having an issue with this test, however:

use strict;
use warnings;
use FindBin qw/$Bin/;
use File::Temp qw/tempfile/;
use lib "$Bin/lib";
use Data::Dumper;

use MyTestHelper;

use Test::More tests => 3;

my $helper = bless {}, 'MyTestHelper';

my $example1 = $helper->get_file('MyTestHelper', 'example1');
chomp $example1;

my $example2 = $helper->get_file('MyTestHelper', 'example2');
chomp $example2;


is $example1, 'foobar[% test_var %]';
is $example2, 'bazquux';

my ($fh, $fn) = tempfile;
$helper->render_file($fn,  { test_var => 'test_val' });
seek $fh, 0, 0; # Rewind
my $contents;
{
    local $/;
    $contents = <$fh>;
}
warn $contents;
is $contents, 'foobartest_val';

The first and second tests pass just fine.  The third one spits out this nice bit of garbage:

devin@devin-laptop:~/web-devel/Catalyst-Devel/1.00/branches/helper_refactor$ prove -l t/back_compat.t
t/back_compat.t .. 1/3 readline() on unopened filehandle DATA at (eval 226) line 1.
Use of uninitialized value $data in split at /home/devin/web-devel/Catalyst-Devel/1.00/branches/helper_refactor/lib/Catalyst/Helper.pm line 59.
Couldn't process "MyTestHelper=HASH(0xcccdf0)", Not a GLOB reference at /usr/local/lib/perl/5.10.0/Template/Provider.pm line 618.
 at t/back_compat.t line 25
# Looks like you planned 3 tests but ran 2.
# Looks like your test exited with 2 just after 2.
t/back_compat.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 1/3 subtests

Test Summary Report
-------------------
t/back_compat.t (Wstat: 512 Tests: 2 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 3 tests but ran 2.
Files=1, Tests=2,  1 wallclock secs ( 0.04 usr  0.00 sys +  0.47 cusr  0.05 csys =  0.56 CPU)
Result: FAIL

Basically, it's telling me it doesn't like this line:

$helper->render_file($fn,  { test_var => 'test_val' });

Why? I've yet to figure it out.  More head pounding is on the way.

Intro:
This document is to get opinions on a new, up to date set up  and refactoring of the Catalyst::Helper API.

What We Have
Currently, all template/image data is hideously store inline with code in the __DATA__ section of Catalyst::Helper.pm.  The API is designed to handle data this way, which is wrong wrong wrong for a number of reasons:

1.  You have to actually delve into Catalyst internals to edit any of these files
2.  If you create a helper, your data must be inline as well.
3.  Its current layout does not reflect the directory hierarchy of a Catalyst application, thus making it pretty dang hard to expand upon/upgrade.

There are no methods to allow you to edit previously created files, unless you want to create your own helper and feverishly create your own file modification method(s).
I can attest to the stress this causes, as I've been doing a good deal of yak shaving cleaning this code up and creating the proper tests for it.  It's not a very fun ordeal at this point.

What we want
From the feedback I've gathered thus far:

We do want previously created helper files to be modifiable via Catalyst::Helpers.  For instance, updating Makefile.PL, updating myapp.conf to reflect changes, updating DBIC schema files to reflect database changes (with minimal pain). Also, add an infrastructure for modifying existing catalyst code, e.g. moving Catalyst::Helper::AuthDBIC from an  experimental proof of concept to something that people might actually want to use for their own stuff.  

We do want to be able to remove a good deal of the boilerplate code that still seems to need to be created manually, for each app, even though it's the exact same code for each instance.

We do want to set up the helper files in the hierarchy that a Catalyst application is created in, thus cutting quickly to the chase in the way of creating an app.  It's a simple name and Template::Toolkit variable translation, then writing the file to the filesystem process then.  This gets our binary data out of the API code, the template data out of the API code, and generally makes every one more happy. 

With this said, here are some more specific ideas that have been passed my way that I think would really help further advance our precious Helper API:

1.  Beat up MooseX::Getopt to deal with pass-through options, so you can deal with passing data structures as arguments easier. Not to mention actually start USING it in the Helper API.
2. Add a feature to write out DBIC schema files for deployment.  Yes, make_schema_at does this, but you have to write your own little script to enjoy this morsel of functionality.  It would be nice to have it packaged up and ready to go for you.
3. Have something like TTSite, but less sucky.  Perhaps there could be a default layout that can be created, or you can create your own sort of skin and have that the default generation.  Or, better yet, packaged skins that users can submit to $somewhere and have the author choose a "theme" for their app. For instance, using $javascript framework and $css framework to do so, like in chapt 11 of the upcoming book (Example usage: http://www.uow.edu.au/~kd21/)

These are the current ideas I have, and that others have submitted.  Sure this RFC is a bit bare, but we don't have a whole ton to work with at the moment with Catalyst::Helper, so we need more feedback to beef things up.

Responses/abuse is more than welcome.