Archive for the ‘PHP’ Category.
The January 2011 issue of php|architect Magazine includes an article by me on a subject that I’ve spoken and written about on several occasions now, features of the Standard PHP Library (SPL) introduced in PHP 5.3. Feel free to leave a comment on this blog post if you read it and enjoy it or have a suggestion on how to improve my presentation of the material. Thanks in advance!
I know a number of my readers have been waiting for this announcement: my book, Web Scraping with PHP, is now available for sale in hard copy form! That’s right, you can now finally order your very own print edition copy. And if you happen to have it with you when next we encounter each other and would like me to autograph it for you, I will be most happy to do so.
To those who felt forced into buying the PDF edition to get access to the content because a print edition was not available until now, you have my most sincere and profound apologies. I wish there was something more I could have done to expedite this. I have inquired with the publisher as to whether it might be possible for you to get a print copy for the difference in what you’ve already paid versus the cost of the Print+PDF bundle. I will update this blog post if I hear any news related to this inquiry.
Again, thanks to everyone for your continued interest, purchases, and support. You’ve made all the time and energy put into this book worthwhile.
Update: If you’ve purchased the PDF-only version of the book and would like a print copy, contact the publisher and they will try to accommodate you.
I was recently writing a unit test for an autoloader when I came across a somewhat unintuitive behavior in PHPUnit.
One requirement of the test suite was that some test methods had to be run in a separate process since class declarations reside in the global scope and persist until the process terminates. So, I slapped a @runInSeparateProcess annotation in the docblock of a test method with that requirement, ran the test suite… and watched that test method fail because the class was still being declared.
It took some head-scratching and tracing through the source of PHPUnit itself to figure out what was going on. When you run the phpunit executable, it’s actually instantiating PHPUnit_TextUI_TestRunner. The eventual result of this is that the run() method inherited by your subclass of PHPUnit_Framework_TestCase is called.
Depending on the value of the also-inherited $preserveGlobalState instance property, which can be set via the setPreserveGlobalState() method, multiple measures are undertaken to preserve the state of the current process. One such measure is including files for all the classes currently defined in that process, which is what was tripping me up because $preserveGlobalState has a default value of true.
$preserveGlobalState must contain its intended value before the run() method is called. The easiest way that I’ve found to facilitate this is to override the run() method in your subclass, call setPreserveGlobalState() there, then call the parent class implementation of run(). I’ve included a code sample below to illustrate this.
class MyTestCase extends PHPUnit_Framework_TestCase
{
public function run(PHPUnit_Framework_TestResult $result = NULL)
{
$this->setPreserveGlobalState(false);
return parent::run($result);
}
}
So, if you try to use the @runInSeparateProcess or @runTestsInSeparateProcesses annotations that PHPUnit offers, be aware that the global state will be preserved by default. You will need to explicitly set it to not be so if running tests in separate processes is to have the effect that you are probably intending.
After a recent release of Phergie, I came across a few issues stemming partly from odd behavior in the PHP FilterIterator class.
First, bug #52559. At the time, I was trying to troubleshoot why the first element seemed to be skipped during iteration by a subclass of FilterIterator. Not knowing that FilterIterator contained no count() method at the time, I tried calling it to get the number of elements in the original array.
Once I discovered the segfault, I had to come up with a short code sample exposing the bug in order to report it. Tracing that segfault to that method call was a bit tedious, but Xdebug helped. Lesson learned: when a segfault occurs, check calls to all methods of native classes to ensure that they actually exist. The --rc option of the PHP CLI binary is particularly useful for this.
Last, bug #52560. This bug was an indirect cause of the first element being skipped during iteration. What actually happens is that null is returned in place of that element. The bug report goes over this in more detail, but the easiest way to work around this bug is to override the FilterIterator constructor in the subclass and call the rewind() method to explicitly reset the iterator position when the class is instantiated, as shown below.
<?php
class MyFilterIterator extends FilterIterator {
public function __construct(Iterator $iterator) {
parent::__construct($iterator);
$this->rewind();
}
}
Several people have asked me the same question recently, so I decided to take a blog post to provide an answer. The question is, “When will ‘Web Scraping with PHP‘ be available in print?” Answering requires a bit of background to paint a full picture of where things are now.
I turned in the manuscript for the book, which had already gone through technical editing, back in April 2009. That’s right, 14 months ago. After going through proofreading and layout, having a cover designed, getting an ISBN, and so forth, the digital version of the book finally went up for sale about three months ago.
At that point, I asked if print copies would be available at php|tek. The publisher sounded unsure, but told me that they would see what they could do. I was later told that the book had gone through printing and that arrangements had been made for a box to be shipped to the hotel on the Monday prior to the conference starting. It didn’t arrive at any point before or during the conference.
After the conference was over (now about two months ago), I was told that the box had finally arrived at the hotel the following Wednesday. I’m not sure how this was known, because the box was never recovered from the hotel to be shipped elsewhere.
The publisher then resolved to request another box from the printer. This box would have to be shipped to someone to proof the printed copies for quality assurance, then the books would go on sale. I was told about this resolution in early June, about five or six weeks ago. I’ve been in touch with the person responsible for proofing the books; they haven’t been received yet.
I have no direct contact with the printer, so all I’ve been able to do is e-mail the publisher repeatedly (six times and counting) since then asking for an update. Through an instant messenger conversation, I was told that a call was supposed to be placed to the printer early last week to determine what the status of things was. So far as I know, that call was never made, or at least I was never told of the result.
I’m as frustrated with this situation as anyone. I want the book to be available to people who want to read it in the format they prefer. If you are in this audience, please contact the publisher to voice your opinion. You can do so by using this contact form on their web site or by sending them a tweet on Twitter. Thank you all very much for your continued support.
Daniel Stenberg, one of the primary authors of the libcurl library on which the PHP cURL extension is based, was kind enough to comment on and clarify a recent blog post of mine regarding web scraping using the PHP and cURL. He later sent me a tweet to invite me to a new mailing list for web scraping enthusiasts just before tweeting a public invitation. In addition to the mailing list itself, the web site also has links to books (including my book) and popular tools related to the subject. I think this is awesome and I encourage anyone with an interest in web scraping, professional or recreational, to join.
Obligatory pitch: Many other useful tidbits like this can be yours by purchasing my book, php|architect’s Guide to Web Scraping with PHP.
I recently wrote a PHP script to scrape data from a .NET application. In the process of developing this script, I noticed something interesting that I thought I’d share. In this case, I was using the cURL extension, but the tip isn’t necessarily specific to that. One thing my script did was submit a POST request to simulate a form submission. The code looked something like the sample below.
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_URL => 'http://...',
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => array(
'field1' => 'value1',
// ...
),
// ...
));
The issue I ran into had to do with a behavior of the CURLOPT_POSTFIELDS setting that’s easy to overlook. This is a segment of its description from the PHP manual page for the curl_setopt() function.
If value is an array, the Content-Type header will be set to multipart/form-data.
If the form being submitted is not set to have an enctype attribute value of multipart/form-data in the form’s markup, .NET returns a 500-level HTTP response with no further information on what causes the error (for security purposes). This presumably happens because it’s expecting one value for the Content-Type request header and getting another.
Setting CURLOPT_HEADER and CURLOPT_VERBOSE to true helped to reveal that this was the issue. The fix is pretty simple: instead of passing the array itself for CURLOPT_POSTFIELDS, pass the result of wrapping it in a call to the http_build_query() function (see its PHP manual page). This converts it to a properly formatted query string, which causes cURL to use the default Content-Type header value of application/x-www-form-urlencoded instead.
Tools like Firebug can help you to examine requests made by a browser. Together with these settings for cURL, you can modify your script’s requests to match those of your browser as closely as possible, making gotchas like this less likely to trip you up.
Note: I’ve written on this topic before, but thought the subject warranted further more detailed discussion and a more comprehensive and up-to-date set of benchmarks. Hence, this post and this presentation. Enjoy.
Update: This post and the benchmarks have been updated for PHP 5.3.4 and Ubuntu 10.10. (12/14/2010)
The SPL, or Standard PHP Library, is an often overlooked extension in the PHP core. It first came on the scene in PHP 5 and a variety of iterators constituted the majority of its initial offerings. Though the iterator offerings were expanded in PHP 5.3, the particularly interesting additions to the SPL were several specialized data structure classes, the foundational concepts for which originate in the field of computer science. In this post, I will provide an overview of these new classes and explain why and when they should be used.
While PHP has several data types, the ones that likely see the most frequent and varied use are strings and arrays. They are the proverbial duct tape and WD-40 of PHP, respectively. Like arrays, SPL data structure classes are used to store composite (i.e. non-scalar) data.
Now, that’s not to say that every instance of an array in existing codebases should be replaced with an SPL container object. There are cases where it’s appropriate to use one over the other. Knowing the difference requires an understanding of how arrays work.
Within the C code that makes up the PHP interpreter, arrays are implemented as a data structure called a hash table or hash map. When a value contained within an array is referenced by its index, PHP uses a hashing function to convert that index into a unique hash representing the location of the corresponding value within the array.
This hash map implementation enables arrays to store an arbitrary number of elements and provide access to all of those elements simultaneously using either numeric or string keys. Arrays are extremely fast for the capabilities they provide and are an excellent general purpose data structure.
In contrast to arrays, SplFixedArray functions more like C arrays or PHP arrays. The maximum number of elements that it may contain is specified upon instantiation. While it is possible to change it later via the setSize() method, this negates the performance advantages of using it: because its size is fixed, it doesn’t need to use a hashing function to resolve the position of elements within the array. It makes sense to use fixed arrays when the number of elements to be stored is known in advance and the elements only need to be accessed by sequential position.
SplFixedArray implements the Iterator, ArrayAccess, and Countable interfaces. Iterator allows it to be iterated using a foreach loop. ArrayAccess provides access to its elements using array syntax where elements are referred to using integer positions beginning at 0 as with enumerated arrays. Countable enables a list to be passed to the count() function like an array.
Aside from the inability to use it in place of arrays with array functions, instances of SplFixedArray function just like arrays for all intents and purposes. It’s even possible to convert them to and from arrays using the toArray() and fromArray() methods respectively. However, it generally makes more sense to use SplFixedArray exclusively for each individual use case.
In computer science, a list is defined as an ordered collection of values. A linked list is a data structure in which each element in the list includes a reference to one or both of the elements on either side of it within the list. The term “doubly-linked list” is used to refer to the latter case. In the SPL, this takes the form of the class SplDoublyLinkedList.
Like SplFixedArray, SplDoublyLinkedList also implements the Iterator, ArrayAccess, and Countable interfaces. In addition to the methods that come with these interface implementations, elements can be added to or removed from the start or end of the list using its push(), pop(), shift() and unshift() methods, which correspond to the array_push(), array_pop(), array_shift(), and array_unshift() functions respectively. Unfortunately, as of PHP 5.3.4, there’s no way to insert an element anywhere in the list other than at the beginning or the end. A feature request has been filed for this. Add a comment or vote to show support for its addition.
The elements at the start and end of the list are accessible via its top() and bottom() methods respectively, which correspond to the reset() and end() functions. Like SplFixedArray, elements can also be accessed arbitrarily by positional index using the array syntax granted by ArrayAccess. It makes sense to use lists when the number of elements to be stored is not known in advance and the elements only need to be accessed by sequential position.
Stacks are similar to lists with two major differences. First, elements can only be added to the top of the stack. Second, an element can only be accessed by taking it off the top of the stack. Because of these differences, the stack is often referred to as a Last-In-First-Out or LIFO data structure. SplStack is the SPL stack implementation.
SplStack is a bit removed from the traditional definition of a stack. It extends SplDoublyLinkedList and inherits its abilities, some of which don’t really apply to stacks. In order to enforce its restriction on how elements are accessed, SplStack overrides the setIteratorMode() method of its parent class and implements its own to prevent modification of the iteration direction. Both methods allow elements to be retained or removed as they are iterated.
Use of stacks makes sense when the number of elements to be stored is not known in advance and the only element that must be accessible is the last one stored. However, as of PHP 5.3.4, the performance of SplStack leaves something to be desired. Benchmarks included later in this provide an objective illustration of this, though the cause of the behavior remains unknown.
Queues are also similar to lists, again with two major differences. First, elements can only be added (or “enqueued”) to the end of the queue. Second, an element can only be accessed by removing (or “dequeueing”) it from the beginning of the queue. For these reasons the queue is referred to as a First-In-First-Out or FIFO data structure. The SplQueue class implements this data structure in the SPL.
SplQueue follows suit with SplStack in extending SplDoublyLinkedList. Just as SplStack resultingly inherits some operations with at least questionable applicability, so too does SplQueue. Likewise, it overrides setIteratorMode() with its own version to restrict how elements are accessed. Use of queues makes sense when the number of elements to be stored is not known in advance and the only element that must be accessible is the remaining element that was stored earliest.
One minor difference between SplQueue and SplStack is that the former contains two method aliases named after conceptual queue operations: dequeue() aliases SplDoublyLinkedList::shift() and enqueue() aliases SplDoublyLinkedList::push(). This makes sense because while push() and pop() share similar applicability to conceptual stack operations, they are already present in its parent class.
Despite their common ancestry, SplQueue appears to have better performance than SplStack as of PHP 5.3.4. Benchmarks included later in this post review this in more detail.
Up to this point, the data structures discussed have resembled lists insofar as they contain elements in the order in which they were added. By contrast, when an element is added to a heap, a comparison function is used to compare the new element to other elements already in the heap and element is placed appropriately within the heap based on that function’s return value. The beauty of heaps is that their underlying algorithm does this with minimal element comparisons, so it’s extremely efficient. Using heaps makes sense when the number of elements to be stored is not known in advance and elements must be accessed in an order based on how they compare to each other.
SplHeap is an abstract class used to create a heap by extending it and providing a comparison function in the form of its compare() method. Only the root element of a heap, the one yielding the highest comparison function return value, may be accessed or removed from the heap at any given time. This is done using the extract() method of SplHeap. SplHeap implements the Iterator and Countable interfaces but, because only the root element can be extracted, it does not implement the ArrayAccess interface like the previously discussed data structure classes.
In addition to the abstract SplHeap class, two concrete implementations are also included in the SPL, namely SplMinHeap and SplMaxHeap. The compare() method of SplMinHeap returns a value such that the smallest element in the heap is the root element. Likewise, the compare() method of SplMaxHeap returns a value such that the largest element in the heap is the root element.
At first glance, using a subclass of SplHeap may seem equivalent to calling sort() or a similar function on an array and accessing the elements in sequence. This is indeed the case if all elements are added to the array prior to it being sorted. However, situations such as elements arriving over time or inadequate memory to store all elements simultaneously may preclude this approach. Use of arrays in such situations would require repeated resorting of the entire array as new elements are added, which is inefficient. This is why using the corresponding heap class makes a lot more sense in that situation than repeated calls to sort(), min() or max(). Additionally, SplHeap can be used to implement the heapsort algorithm, which has better worst case performance than the quicksort algorithm implementation used by arrays.
Priority queues are somewhat similar to heaps. In fact, while it doesn’t extend SplHeap, SplPriorityQueue does make use of a heap structure internally to implement its functionality. The difference is that the insert() method of SplPriorityQueue queue accepts both a value and an associated priority, removing the need to use an array or object to store both of these and define an appropriate comparison function in an SplHeap instance. Elements with the highest priority, like those in SplMaxHeap with the highest value, are the ones that come out first when extract() is called. Note that elements with equal priority are returned in no particular order.
For reasons similar to those of SplHeap, SplPriorityQueue implements both Iterator and Countable interfaces and does not implement the ArrayAccess interface. Because it stores a value and priority per element, SplPriorityQueue includes a setExtractFlags() method that modifies the behavior of extract() to return the stored value, the stored priority, or an array containing both. Priorities are not bound to a particular data type: strings, integers, or even composite data types can be used. SplPriorityQueue can be extended and its compare() method overridden to customize the comparison logic.
It makes sense to use a priority queue when the number of elements to be stored is not known in advance and elements must be accessed in an order based on how a value associated with each element (versus the element value itself) compares to the same associated values of other elements.
SplObjectStorage combines some of the properties of two different data structures. First, it provides the same functionality of a hash table that a normal array has, but without its associated inability to use objects as keys unless the spl_object_hash() function is used. In other words, it implements a composite hash map. Second, it can be used as a set to store objects as data without a meaningful corresponding key or concept of sequential order.
Its attach() method accepts an object key and the data to associate with it and its detach() method allows data to be removed using its associated object key. To use the object as a set, simply exclude the $data parameter for attach() as it’s optional. The set operations implemented by SplObjectStorage all have array function counterparts. For example, the addAll() method and array_merge() function both correspond to the union set operation. The difference operation is available using the removeAll() method and array_diff() function and its variants. The contains() method and in_array() function both implement the element_of operation. Sadly, only arrays have an implementation of the intersection operation in the form of array_intersect() and its variants. Tobias Schlitt has a more in-depth analysis of this data structure that includes implementations of the set operations lacking in the SPL itself.
Update: A patch I’ve submitted has been merged. SplObjectStorage::removeAllExcept(), which is equivalent to the set intersection operation, will become available in PHP 5.3.5. (1/5/2011)
Like some of the other data structures in the SPL, SplObjectStorage implements the Iterator, Countable, and ArrayAccess interfaces. Oddly, it also implements the Traversable interface (which is limited to internally defined classes and negates the need for implementation of the Iterator interface) and the Serializable interface (and it is the only SPL data structure class to do so).
Using this class makes sense when data must be stored using composite keys or the ability to access data using set operations is more important than accessing data in a specific order.
Standard disclaimer: There are lies, damned lies, and benchmarks. YMMV.
--without-pear --without-sqlite --without-sqlite3 --without-pdo-sqliteCode used is located in this GitHub repository.
 |
 |
CodeArray |
 |
 |
CodeArray |
 |
 |
CodeArray |
 |
 |
CodeArray |
 |
 |
CodeArray |
 |
 |
CodeArray |
 |
 |
CodeArray |
If you have an interest in other data structure implementations for PHP outside of SPL offerings, check out the bloomy PECL extension, which is an implementation of a bloom filter created by Andrei Zmievski.
What I’m announcing in this blog post has been in the works since early 2008 when I first pitched the idea. It was rejected by several major publishers who basically said the same thing: the idea was in too small of a niche or simply wasn’t marketable. php|architect Press respectfully disagreed with them and decided to publish what is now a book written by me that you can purchase.
It’s currently only available in PDF format due to a delay with the printer; a dead tree version should become available within the next few weeks. To my knowledge, there are plans to offer the paper and PDF bundle as has been done in the past with their other books.
Many of you reading this post probably have a personal to-do list of goals that you want to accomplish within your lifetime. Becoming the published author of a book has been an item on my own list for some time, one that seeing this accomplishment through to its completion has helped me to cross out. I think anyone who has achieved a similar victory can relate to its significance, if only to oneself.
I do of course encourage you to purchase the book. I have no naïve notions that this will result in any substantial monetary return. Even if it did, that was not my reason for writing the book. I did it because I have knowledge that I believe is worth knowing and sharing with you. There were a number of people who contributed to this and I encourage you to read about them in the pages of the book that credit them.
It is also worth restating here that I have many family members, friends, and colleagues who helped to make this possible. There are too many to name, but I would like to thank each and every one of you from the bottom of my heart. I consider this a milestone in my life and my only hope is that it has as profound an effect on your life as it has on my own.
A question that frequently comes up in my interactions with other developers about Zend Framework is how to approach designing models. There’s a small collection of resources and advice that I generally give on the subject, so I thought I’d write up a blog post to give people an easy place to access it all.
First, there is no one “correct” way to design a model. If there were, the framework would probably have an actual model component. It doesn’t, and as Bill Karwin — former Project Manager for Zend Framework — has said, the reason for this because designing the model is your job. There are pros and cons to any approach. It’s all about finding a method that works for you, is appropriate for the situation at hand, and mitigates difficulty in long-term application maintenance.
The Model section of the Quick Start Guide and this blog post by Michelangelo van Dam includes examples of a Data Mapper approach. Zend_Db uses a Table Data Gateway and Row Data Gateway approach. Doctrine, a popular ORM library that is gaining traction in the ZF community, uses an Active Record approach. And available approaches don’t stop there. As a general rule of thumb when designing models and components in general, I recommend favoring composition over inheritance. You’ll get a better sense of what I mean by that later in the post.
The Wikipedia article on the MVC architectural pattern isn’t all-encompassing, but isn’t a bad place to start either. In particular, it drives home a few important points about the model that you should bear in mind.
The model is a “domain-specific representation of the data upon which the application operates.”
In a nutshell, the model handles data: storing it, retrieving it, filtering and validating it, and providing access to it.
The model contains “domain logic” that “adds meaning to raw data.”
In other words, it handles the conversion from raw data in a data source to semantically meaningful PHP objects and back again.
“MVC does not specifically mention the data access layer because it is understood to be underneath or encapsulated by the model.”
The shorter version: $model != $database. This is one point that trips a lot of people up. The model and your database are not congruent, synonymous, or in any way equivalent. Yes, 99% of the time, your model will use a database for its data source. However, models can be more complex than that: they can serve as clients to web services, limit access to data using an ACL, access data caching resources like memcached or APC, and so forth.
“So which approach do you use?”
Generally, the answer is none of the above. My personal preference is to keep data, in the form of plain old PHP arrays and objects, separate from logic to handle that data. Many common PHP tasks result in data already being present in either of these forms such as in its superglobal arrays, so it seems natural to just take it in the form in which it’s provided.
I define a model class that composes some other object to access the data I need, generally a Zend_Db_Adapter instance. Any methods of that model class return data using scalar types or classes that PHP supports natively. What’s great about this is that it’s fairly easy to convert data to and from these forms using type juggling regardless of the data’s origin.
“But wait, how can I encapsulate the data source within my model if I need a dependency like a
Zend_Db_Adapterinstance for it to be able to interact with that data source?”
This is another major question that people tend to ask. If any code calling your model first has to handle injecting its dependencies, that muddies up separation of concerns because calling code then must have some knowledge of how your model operates internally. This is a problem because, if the data source of the model needs to change in the future, all calling code needs to change as well versus only model code. There are a few ways to approach this problem in Zend Framework.
The first method involves storing your dependencies in Zend_Registry. Declare accessor methods in the model for dependencies that retrieve them from the registry only if they are not explicitly injected from calling code. This bypasses the need for dependency injection from application code, thus preserving separation of concerns, but still allows injection to be performed for unit testing purposes.
The second method is a variation of the first and is specific to the case of Zend_Db_Adapter instances. This approach involves setting your adapter as the default adapter to use for Zend_Db_Table instances in lieu of using the registry to store it. Note that this doesn’t require actually using Zend_Db_Table in order to work. This can be set from the bootstrap or application configuration file, then retrieved using the getDefaultAdapter() method of the Zend_Db_Table_Abstract class. Again, don’t forget your accessor methods so dependencies can be injected from unit tests.
Another method might be to use a service locator as a dependency in models. This serves as a layer of abstraction between models and their dependencies when it comes to controllers dealing with both. To my knowledge, the closest thing to an implementation of this in ZF is the Zend_Application_Bootstrap classes. From a controller, the bootstrap instance can be accessed as in the code example below.
$bootstrap = $this->getInvokeArg('bootstrap');
Service layers are another frequent topic related to models. Those aside, I mainly suggest picking a simple model that you can prototype and try several approaches to see which you prefer. If you’ve got some ZF experience under your belt, I’d be interested in hearing about your own modeling approaches and experiences and encourage you to leave a comment on this post.
Photo Credit: Rob Allen
PHP developer, Acadiana Open Source Group organizer, creator of Phergie, and web scraping aficionado
