Not logged in | Login

My Security Planet » Tags » Perl

Feeds

13620 items (0 unread) in 75 feeds

WebSecurity

Security

SEO

SEO

Tools

Tools

Opensourcetesting.org News

8 items tagged "Perl"

Jeremy Zawodny's blog

Pass the Duct Tape

Posted: September 23rd, 2009, 11:40pm CDT

Tags: Perl [edit]

Reading The Duct Tape Programmer really hit a nerve with me. If you code for a living (or part of your living), go read it. I sometimes get annoyed by Joel Spolsky's writing, but this time he hit the nail right on the head.

I am by no means putting myself in the same league as Jamie Zawinski, but I definitely subscribe to the idea that keeping my code as simple as possible is going to pay big, big dividends in the long run.

Needless complexity really does kill you.

In my work it has manifested itself in a few ways. But one that seems to pop up over and over in recent times (and has bitten a few of my coworkers as well) is socket programming. For whatever reason, we're able to find the the freely available client libraries for interacting with some services will inevitably fail a small but non-zero percentage of the time. And when you're making millions and millions of calls or connections per day, even a 0.01% failure rate is enought to make for a really bad day.

That's especially true when the failure mode involves leaking file descriptors or needlessly long timeouts (or timeouts that fail to work properly) that result in a full or partial cascading failure. It's the ultimate in frustration.

I'm not going to name names here. Maybe at an upcoming conference I'll talk about what we've seen and how network oddities add a whole new set of failures to the mix.

Anyway, the solution ends up being one of two options: (1) forking the module and re-writing the code to use the low-level system calls instead of the multitude of abstraction layers that were supposed to make the task easier (die abstraction layer! die!), or (2) scraping the module entirely and writing your own. The first option sucks but at least you're not having to learn all the quirks of those abstraction layers. The second option sucks even more but at least you know exactly what you've got when it's all said and done.

For me it's more satisfying and productive get to know the low-level stuff well enough that you can just drop it in when you need it. It's less frustrating than trying to reproduce all the conditions and variable that cause some abstraction layer to fail.

Pass the duct tape. I've got work to do!

(comments)
Bash Trick: Watching Multiple Background Jobs

Posted: November 21st, 2008, 9:21am CST

Tags: Perl [edit]

I recently had a need to add some error checking to a bash script that runs multiple copies of a Perl script in parallel to better utilize a multi-core server. I wanted a way to run these four processes in the background and gather up their exit values. Then, if any of them failed, I'd prematurely exit the bash script and report the error.

After a bit of reading bash docs, I came across some built-ins that I hadn't previously used or even seen. First, I'll show you the code:
wait.sh
This is the bash script that runs the parallel processes and gathers up the exit values.
#!/bin/bash FAIL=0 echo "starting" ./sleeper 2 0 & ./sleeper 2 1 & ./sleeper 3 0 & ./sleeper 2 0 & for job in `jobs -p` do echo $job wait $job || let "FAIL+=1" done echo $FAIL if [ "$FAIL" == "0" ]; then echo "YAY!" else echo "FAIL! ($FAIL)" fi sleeper
And here's the Perl script that I wrote in order to test the functioning of wait.sh. It accepts to arguments. The first is the number of seconds to sleep (to simulate the delay associated with doing work) and the second is the exit value it should use (any non-zero value indicates a failure).
#!/usr/bin/perl -w use strict; my $time = $ARGV[0] || 1; my $exit = $ARGV[1] || 0; sleep $time; exit $exit; Discussion
New to me was the use of let to do math on a variable so that I can count up the number of failures. Is there a better way? There's no native ++ operator in bash. Similarly, using jobs to get a list of pids to wait on provided to be a very useful idiom.

The code is straightforward and works for my purposes. But since 99% of my time is spent in Perl rather than bash, I wonder what I could have done differently and/or better. Feedback welcome.

And, if this is at all useful to you, feel free to take it and run...

Finally, I'm starting to really dig gist.github for showing off bits of code. It's good stuff.

(comments)
Asynchronous MySQL Client in Perl

Posted: November 14th, 2008, 9:47am CST

Tags: Perl [edit]
I recently found myself wishing for an async library for MySQL. My goal is to be able to fire off queries to a group of federated servers in parallel and aggregate the results in my code.

With the standard client (DBD::mysql), I'd have to query the servers one at a time. If there are 10 servers and each query takes 0.5 seconds, my code would stall for 5 seconds. But by using an async library, I could fire off all the queries and fetch the results as they become available. The overall wait time should not be much more than 0.5 seconds.

While I found little evidence of anyone doing this in practice, my search led me to the perl-mysql-async project on Google Code. It's a pure-Perl implementation of the MySQL 4.1 protocol and an asyncronous client that uses Event::Lib (and libevent) under the hood.

The code contains little in the way of documentation or examples, aside from the simple bundled test script. After a bit of mucking around with it, I managed to cobble together a working example. It looks like this:
```
#!/usr/bin/perl -w
use strict;

use Event::Lib;
use Data::Dumper;

use MysqlAsync;
use AsyncCaller qw/schedule/;

$Data::Dumper::Terse = 1;

$|=1;

my $expected_results = 25;
my $results = 0;

my $dbh;

for (1..$expected_results) {
        # my $secs  = int(rand(5));
        my $secs  = rand(5);
        my $query = qq[select sleep($secs)];
        schedule(0.001, sub{
            my $dbh = MysqlAsync->new(
            database => {
                host => "localhost",
                port => 3306,
                database => "mysql",
                passwd => "xxxxxx",
                user => "root",
            },
            connect_timeout => 1,
            max_requests    => 25,
            db_timeout      => 10,
            # logfile         => "/tmp/mysqllog",
        );

        $dbh->get_array($query, &result );
        });
}

event_mainloop();

exit;

sub result
{
    my ($result) = @_;
    if (defined $result) {
        print "result: " . Dumper($result);
    } else {
        print "error: " . Dumper($dbh->error());
    }
    $results++;

    # all done?
    if ($results == $expected_results) {
        exit;
    }
}

__END__
```
Sure enough, that code runs in just a bit more time than the longest query it executes, rather than the sum of all the query times.

What still surprises me is that this code doesn't appear to get a lot of use (or at least discussion) in the real world. In the PHP world, the mysqlnd driver offers async queries.

So count this as my contribution to demonstrating that Perl can do async MySQL queries too.

(comments)
Programming Annoyance: Libraries that Exit on Me

Posted: October 1st, 2008, 10:45am CDT

Tags: Perl [edit]
This is something that's been bugging me for a long time now. Over the years, I've come to realize that programming time is 10% about writing the code to do the work, 70% about figuring out where failures might occur and dealing with them, 10% about documentation, and 10% about documentation. (That last 10% may be substituted with Desktop Tower Defense or something equally time wasting.)

Or something like that. The point is that writing the code to do what I want isn't hard. It's dealing with all the other things that do--especially error conditions. There are so many weird corner cases to consider. And when you're working on code for a high volume web site that has its servers under load 24 hours a day, it doesn't take long to encounter those odd situations.

Murphy is always watching.

Years ago, after battling similar problems at Yahoo, I began to develop certain ideas about how errors should be detected, handled, and reported. An important idea here is that the developer should always be in control of when the script/program/process dies. Aside from something truly fatal (like a segfault) library routines should detect errors and report them back to their caller in the form of a known-to-be-bad return value.

The problem is that I keep running into code I want to use that breaks that rule in multiple places. In Perl terms, that means that I'll be happily testing my code and suddenly something goes wrong and my script dies in a place I didn't expect. Upon digging into it, I find that the CPAN library I'm using has something like this lurking in it:
```
if (not $good) {
    Carp::croak("bad stuff happened!");
}
```
Or...
```
if (not $good) {
    die "badness here!";
}
```
Sigh!

This means I have to read the code a bit more and see if I can discern why the developer wants my script to die in some cases, but in others he's content to just do this:
```
if (not $good) {
    $@ = "bad things happened";
    return undef;
}
```
What is it about some errors that makes them fatal while others aren't so bad that I'm deemed able to deal with them? Why has this developer taken that decision away from me? It makes no sense at all.

What this means is that I then need to litter my code with ugly crap like this:
```
eval {
    $object->methodThatMayDie;
};
if ($@) {
    # handle error here
}
```
The problem with that, aside from the fact that I'm dealing with another developer's inconsistent coding, is that it pollutes my code and forces me to make yet another frustrating decision.

Do I use a small number of big eval blocks and give up knowing exactly where the code died? Or do I pollute my code with a larger number of smaller eval blocks so that I can react to specific problems with a more specific solution? That means the module developer would have had to document which methods or functions may die on me. Otherwise I have to go trudging through their code and waste my time figuring that out. Guess which is more frequent.

Or do I override the module's use of die or Carp or whatever. I can do it, but that has other side effects I probably don't want to deal with either.

Why do I even need to deal with this in the first place? Can't people provide consistent interfaces? Is there something so bad about returning an error code and leaving it up to the user of your code to decide how to handle error conditions?

Maybe they do want to exit() or die(). Maybe they want to retry the logic after waiting a bit. Maybe they want to page someone and log the failure. Maybe...

You get the idea.

This whole concept of "fatal" exceptions seems wrong to me. Unless things are so bad that the kernel is going to kill my process, I should be the one in charge of deciding when my code will blow up. And I shouldn't have to do extra work to asset that authority. Should I?

I know that in the Java world, it's common to do a bunch of stuff in a big try block and then try to figure out what, if anything, blew up later. But I'm a firm believer in dealing with specific problems at the exact place they occur.

I really wish more people thought that way. It'd make my life easier.

(comments)
The Perl UTF-8 and utf8 Encoding Mess

Posted: September 2nd, 2008, 5:10pm CDT

Tags: Perl [edit]
I've been hacking on some Perl code that extracts data that comes from web users around the world and been stored into MySQL (with no real encoding information, of course). My goal it to generate well-formed, valid XML that can be read by another tool.

Now I'll be the first to admit that I never really took the time to like, understand, or pay much attention to all the changes in Perl's character and byte handling over the years. I'm one of those developers that, I suspect, is representative of the majority (at least in this self-centered country). I think it's all stupid and complicated and should Just Work... somehow.

But at the same time I know it's not.

Anyway, after importing lots of data I came across my first bug. Well, okay... not my first bug. My first bug related to this encoding stuff. The XML parser on the receiving end raised hell about some weird characters coming in.

Oh, crap. That's right. This is the big bad Internet and I forgot to do anything to scrub the data so that it'd look like the sort of thing you can cram into XML and expect to maybe work.

A little searching around managed to jog my memory and I updated my code to include something like this:
```
  use Encode;

  ...

  my $data = Encode::decode('utf8', $row->{'Stuff'});
```
And all was well for quite some time. I got a lot farther with that until this weekend when Perl itself began to blow up on me, throwing fatal exceptions like this:
```
  Malformed UTF-8 character (fatal) ...
```
My first reaction, like yours probably, was WTF?!?! How on god's green earth is this a FATAL error?

After much swearing, a Twitter plea, and some reading (thanks Twitter world!), I came across a section of the Encode manual page from Perl.

I'm going to quote from it a fair amount here because I know you're as lazy as I am and won't go read it if I just link here. The relevant section is at the very end (just before SEE ALSO) and titled UTF-8 vs. utf8.
```
    ....We now view strings not as sequences of bytes, but as
    sequences of numbers in the range 0 .. 2**32‐1 (or in the case of
    64‐bit computers, 0 .. 2**64‐1) ‐‐ Programming Perl, 3rd ed.

  That has been the perls notion of UTF−8 but official UTF−8 is more
  strict; Its ranges is much narrower (0 .. 10FFFF), some sequences
  are not allowed (i.e. Those used in the surrogate pair, 0xFFFE, et
  al).

  Now that is overruled by Larry Wall himself.

    From: Larry Wall 
    Date: December 04, 2004 11:51:58 JST
    To: perl‐unicode@perl.org
    Subject: Re: Make Encode.pm support the real UTF‐8
    Message‐Id: 

    On Fri, Dec 03, 2004 at 10:12:12PM +0000, Tim Bunce wrote:
    : Ive no problem with utf8 being perls unrestricted uft8 encoding,
    : but "UTF‐8" is the name of the standard and should give the
    : corresponding behaviour.

    For what its worth, thats how Ive always kept them straight in my
    head.

    Also for what its worth, Perl 6 will mostly default to strict but
    make it easy to switch back to lax.

    Larry

  Do you copy?  As of Perl 5.8.7, UTF−8 means strict, official UTF−8
  while utf8 means liberal, lax, version thereof.  And Encode version
  2.10 or later thus groks the difference between "UTF−8" and "utf8".

    encode("utf8",  "x{FFFF_FFFF}", 1); # okay
    encode("UTF‐8", "x{FFFF_FFFF}", 1); # croaks

  "UTF−8" in Encode is actually a canonical name for "utf−8−strict".
  Yes, the hyphen between "UTF" and "8" is important.  Without it
  Encode goes "liberal"

    find_encoding("UTF‐8")‐>name # is utf‐8‐strict
    find_encoding("utf‐8")‐>name # ditto. names are case insensitive
    find_encoding("utf8")‐>name  # ditto. "_" are treated as "‐"
    find_encoding("UTF8")‐>name  # is utf8.
```
Got all that?

The sound you heard last night was me banging my head on a desk. Repeatedly.

I mean, how could I have possibly noticed the massive difference between utf8 and UTF-8? Really. I must have been on some serious crack.

Sigh!

Needless to say my code now looks more like this:
```
  use Encode;

  ...

  my $data = Encode::decode('UTF-8', $row->{'Stuff'}); ## fuck!
```
Actually, I was kidding about the "fuck!" I wouldn't swear in code.

(comments)
The Perl UTF-8 and utf8 Encoding Mess

Posted: September 2nd, 2008, 5:10pm CDT

Tags: Perl [edit]
I've been hacking on some Perl code that extracts data that comes from web users around the world and been stored into MySQL (with no real encoding information, of course). My goal it to generate well-formed, valid XML that can be read by another tool.

Now I'll be the first to admit that I never really took the time to like, understand, or pay much attention to all the changes in Perl's character and byte handling over the years. I'm one of those developers that, I suspect, is representative of the majority (at least in this self-centered country). I think it's all stupid and complicated and should Just Work... somehow.

But at the same time I know it's not.

Anyway, after importing lots of data I came across my first bug. Well, okay... not my first bug. My first bug related to this encoding stuff. The XML parser on the receiving end raised hell about some weird characters coming in.

Oh, crap. That's right. This is the big bad Internet and I forgot to do anything to scrub the data so that it'd look like the sort of thing you can cram into XML and expect to maybe work.

A little searching around managed to jog my memory and I updated my code to include something like this:
```
  use Encode;

  ...

  my $data = Encode::decode('utf8', $row->{'Stuff'});
```
And all was well for quite some time. I got a lot farther with that until this weekend when Perl itself began to blow up on me, throwing fatal exceptions like this:
```
  Malformed UTF-8 character (fatal) ...
```
My first reaction, like yours probably, was WTF?!?! How on god's green earth is this a FATAL error?

After much swearing, a Twitter plea, and some reading (thanks Twitter world!), I came across a section of the Encode manual page from Perl.

I'm going to quote from it a fair amount here because I know you're as lazy as I am and won't go read it if I just link here. The relevant section is at the very end (just before SEE ALSO) and titled UTF-8 vs. utf8.
```
    ....We now view strings not as sequences of bytes, but as
    sequences of numbers in the range 0 .. 2**32‐1 (or in the case of
    64‐bit computers, 0 .. 2**64‐1) ‐‐ Programming Perl, 3rd ed.

  That has been the perls notion of UTF−8 but official UTF−8 is more
  strict; Its ranges is much narrower (0 .. 10FFFF), some sequences
  are not allowed (i.e. Those used in the surrogate pair, 0xFFFE, et
  al).

  Now that is overruled by Larry Wall himself.

    From: Larry Wall 
    Date: December 04, 2004 11:51:58 JST
    To: perl‐unicode@perl.org
    Subject: Re: Make Encode.pm support the real UTF‐8
    Message‐Id: 

    On Fri, Dec 03, 2004 at 10:12:12PM +0000, Tim Bunce wrote:
    : Ive no problem with utf8 being perls unrestricted uft8 encoding,
    : but "UTF‐8" is the name of the standard and should give the
    : corresponding behaviour.

    For what its worth, thats how Ive always kept them straight in my
    head.

    Also for what its worth, Perl 6 will mostly default to strict but
    make it easy to switch back to lax.

    Larry

  Do you copy?  As of Perl 5.8.7, UTF−8 means strict, official UTF−8
  while utf8 means liberal, lax, version thereof.  And Encode version
  2.10 or later thus groks the difference between "UTF−8" and "utf8".

    encode("utf8",  "x{FFFF_FFFF}", 1); # okay
    encode("UTF‐8", "x{FFFF_FFFF}", 1); # croaks

  "UTF−8" in Encode is actually a canonical name for "utf−8−strict".
  Yes, the hyphen between "UTF" and "8" is important.  Without it
  Encode goes "liberal"

    find_encoding("UTF‐8")‐>name # is utf‐8‐strict
    find_encoding("utf‐8")‐>name # ditto. names are case insensitive
    find_encoding("utf8")‐>name  # ditto. "_" are treated as "‐"
    find_encoding("UTF8")‐>name  # is utf8.
```
Got all that?

The sound you heard last night was me banging my head on a desk. Repeatedly.

I mean, how could I have possibly noticed the massive difference between utf8 and UTF-8? Really. I must have been on some serious crack.

Sigh!

Needless to say my code now looks more like this:
```
  use Encode;

  ...

  my $data = Encode::decode('UTF-8', $row->{'Stuff'}); ## fuck!
```
Actually, I was kidding about the "fuck!" I wouldn't swear in code.

(comments)
Fun with Network Programming, race conditions, and recv() flags

Posted: August 7th, 2008, 11:42pm CDT

Tags: Perl [edit]
Last week I had the opportunity to do a bit of protocol hacking and found myself stymied by what seemed like a race condition. As with most race conditions, it didn't happen often--anywhere from 1 in 300 to 1 in 5,000 runs. But it did happen and I couldn't really ignore it.

So I did what I often do when faced with code that's doing seemingly odd things: insert lots of debugging (otherwise known as "print statements"). Since I didn't know if the bug was in the client (Perl) or server (C++), I had to instrument both of them. I'd changed both of them a bit, so they were equally likely in my mind.

Well, to make a long, boring, and potentially embarrassing story sort, I soon figured out that the server was not at fault. The changes I made to the client were the real problem.

I had forgotten about how the recv() system call really works. I had code that looked something like this (in Perl):
```
recv($socket, $buffer, $length, 0);
...
if (length($buffer) != $length) {
    # complain here
}
```
The value of $length was provided by the server as part of its response. So the idea was that the client would read exactly $length bytes and then move on. If it read fewer, we'd be stuck checking again for more data. And if we did something like this:
```
while (my $chunk = ) {
    $buffer .= $chunk;
}
```
There's a good chance it could block forever and end up in a sort of deadlock, each waiting for the other to do something. The sever would be waiting for the next request and the client would be waiting for the sever to be "done."

Unfortunately for me, the default behavior of recv() is not to block. That means the code can't get stuck there--it simply does a best effort read. If you ask for 2048 bytes to be ready but only 1536 are currently available, you'll end up with 1536 bytes. And that's exactly the sort of thing that'd happen every once in a while.

The MSG_WAITALL flag turned out to be the solution. You can probably guess what it does...

This flag requests that the operation block until the full request is satisfied. However, the call may still return less data than requested if a signal is caught, an error or disconnect occurs, or the next data to be received is of a different type than that returned.

That's pretty much exactly what I wanted in this situation. I'm willing to handle the signal, disconnect, and error cases. Once I made that change, the client and server never missed a beat. All the weird debugging code and attempts to "detect and fix" the problem were promptly ripped out and the code started to look correct again.

The moral of this story is that you should never assume that the default behavior is what you want. Check those flags.

Now don't get me started about quoting and database queries...

(comments)
Perl as a Web Scripting Language

Posted: January 25th, 2008, 3:02pm CST

Tags: Perl [edit]

Had I not been at work, I might have nearly fallen off my seat laughing as I read item #3 in You Used Perl to Write WHAT?! on page 2. You see, the author is taking about where Perl is good and bad, and he says "there are some uses that just aren't right." That' followed by a list of items. Here's number three.

As a Web scripting language: One of the earliest usages of perl, as the Web evolved, was for CGI programming webpages. As a result, perl has some pretty strong packages for dealing with Web forms. There is also support for embedding perl into HTML in the same way the Java is embedded into JSP pages.

However, I would argue that more modern Web scripting languages, such as PHP and Ruby on Rails, offer more out-of-the-box Web support and a cleaner integration into the webpage experience. You should especially avoid using perl for traditional CGI-style form processing; this code tends to be hard to read and maintain because the HTML ends up inlined inside the perl code.

Uhm, WHAT?!

Now maybe it's the fact that I started coding in Perl about 12 years ago. Or maybe it's the fact that I've used it quite successfully in various web projects over the years. But suggesting that Perl isn't a good web scripting language is rather laughable.

Seriously.

Now, if you're starting from scratch and looking to learn a new language primarily for building web applications, that's one thing. Sure, look at PHP, Ruby, Python, and whatever else can do the job. But the author seems to be saying that it's worth learning yet another language simply for building web applications.

That's messed up.

Have a look around CPAN and at the web frameworks that exist around Perl and mod_perl. Heck, some of them have been around a long time and are quite mature. Saying that Perl "has some pretty strong packages for dealing with Web forms" doesn't really capture the true state of things.

I think programming language battles are as dumb as the next person, so I generally try to avoid them. But when I see stuff like this, it's hard to keep my mouth shut.

As for the claim that traditional CGI-style form processing being hard to read and maintain... WHAT?! There's nothing hard about separating the HTML from the code.

Okay, enough of my griping...

Thanks to the folks at TechCzar for translating my tech blog posts and including them in their blog network.

(comments)