ViPlus.BlogSpot.Com: Methods

Showing posts with label Methods. Show all posts

Do-It-Yourself Caching Methods With WordPress

There are different ways to make your website faster: specialized plugins to cache entire rendered HTML pages, plugins to cache all SQL queries and data objects, plugins to minimize JavaScript and CSS files and even some server-side solutions.

But even if you use such plugins, using internal caching methods for objects and database results is a good development practice, so that your plugin doesn’t depend on which cache plugins the end user has. Your plugin needs to be fast on its own, not depending on other plugins to do the dirty work. And if you think you need to write your own cache handling code, you are wrong. WordPress comes with everything you need to quickly implement varying degrees of data caching. Just identify the parts of your code to benefit from optimization, and choose a type of caching.

WordPress implements two different caching methods:

Non-persistent
The data remains in the cache during the loading of the page. (WordPress uses this to cache most database query results.)Persistent
This depends on the database to work, and cached data can auto-expire after some time. (WordPress uses this to cache RSS feeds, update checks, etc.)

When you use functions such as get_posts() or get_post_meta(), WordPress first checks to see whether the data you require is cached. If it is, then you will get data from the cache; if not, then a database query is run to get the data. Once the data is retrieved, it is also cached. A non-persistent cache is recommended for database results that might be reused during the creation of a page.

The code for WordPress’ internal non-persistent cache is located in the cache.php file in the wp-includes directory, and it is handled by the WP_Object_Cache class. We need to use two basic functions: wp_cache_set() and wp_cache_get(), along with the additional functions wp_cache_add(), wp_cache_replace(), wp_cache_flush() and wp_cache_delete(). Cached storage is organized into groups, and each entry needs its own unique key. To avoid mixing with WordPress’ default data, using your own unique group names is best.

For this example, we will a create function named d4p_get_all_post_meta(), which will retrieve all meta data associated with a post. This first version doesn’t involve caching.

function d4p_get_all_post_meta($post_id) { global $wpdb; $data = array(); $raw = $wpdb->get_results( "SELECT meta_key, meta_value FROM $wpdb->postmeta WHERE post_id = $post_id", ARRAY_A ); foreach ( $raw as $row ) { $data[$row['meta_key']][] = $row['meta_value']; } return $data;}

Every time you call this function for the same post ID, an SQL query will be executed. Here is the modified function that uses WordPress’ non-persistent cache:

Here, we are using a cache group named d4p_post_meta, and post_id is the key. With this function, we first check to see whether we need any data from the cache (line 4). If not, we run the normal code to get the data and then add it to the cache in line 13. So, if you call this function more than once, only the first one will run SQL queries; all other calls will get data from the cache. We are using the wp_cache_add function here, so if the key-group combination already exists in the store, it will not be replaced. Compare this with wp_cache_set, which will always overwrite an existing value without checking.

As you can see, we’ve made just a small change to the existing code but potentially saved a lot of repeated database calls during the page’s loading.

Non-persistent cache is available only during the loading of the current page; once the next page loads, it will be blank once again.The storage size is limited by the total available memory for PHP on the server. Do not store large data sets, or you might end up with an “Out of memory” message.Using this type of cache makes sense only for operations repeated more than once in the creation of a page.It works with WordPress since version 2.0.

This type of cache relies on a feature built into WordPress called the Transient API. Transients are stored in the database (similar to most WordPress settings, in the wp_options table). Transients need two records in the database: one to store the expiration time and one to store the data. When cached data is requested, WordPress checks the timestamp and does one of two things. If the expiration time has passed, WordPress removes the data and returns false as a result. If the data has not expired, another query is run to retrieve it. The good thing about this method is that the cache persists even after the page has loaded, and it can be used for other pages for as long as the transient’s expiration time has not passed.

If your database queries are complex and/or produce results that might not change often, then storing them in the transient cache is a good idea. This is an excellent solution for most widgets, menus and other page elements.

Let’s say we wanted an SQL query to retrieve 20 posts from the previous month, along with some basic author data such as name, email address and URL. But we want posts from only the top 10 authors (sorted by their total number of posts in that month). The results will be displayed in a widget.

When tested on my local machine, this SQL query took 0.1710 seconds to run. If we had 1000 page views per day, this one query would take 171 seconds every 24 hours, or 5130 seconds per month. Relatively speaking, that is not much time, but we could do much better by using the transient cache to store these results with an expiration time of 30 days. Because the results of this query will not change during the month, the transient cache is a great way to optimize resources.

Returning to my local machine, the improved SQL query to get data from the transient cache is now only 0.0006 seconds, or 18 seconds per month. The advantage of this method is obvious in this case: we’ve saved 85 minutes each month with this one widget. Not bad at all. There are cases in which you could save much, much more (such as with very complex menus). More complex SQL queries or operations would further optimize resources.

Let’s look at the actual code, both before and after implementing the transient cache. Below is the normal function to get the data. In this example, the SQL query is empty (because it is long and would take too much space here), but the entire widget is linked to at the end of this article.

function d4p_get_query_results() { global $wpdb; $data = $wpdb->get_results(' // SQL query // '); return $data;}

And here is the function using the transient cache, with a few extra lines to check whether the data is cached.

function d4p_get_query_results() { global $wpdb; $data = get_transient('my_transient_key'); if ($data === false) { $data = $wpdb->get_results(' // SQL query // '); set_transient('my_transient_key', $data, 3600 * 24); } return $data;}

The function get_transient (or get_site_transient for a network) needs a name for the transient record key. If the key is not found or the record has expired, then the function will return false. To add a new transient cache record, you will need the record key, the object with the data and the expiration time (in seconds), and you will need to use the set_transient function (or set_site_transient for a network).

If your data changes, you can remove it from the cache. You will need the record key and the delete_transient function (or delete_site_transient for a network). In this example, if the post in the cache is deleted or changed in some way, you could delete the cache record with this:

delete_transient('my_transient_key');The theoretical maximum size of data you can store in this way is 4 GB. But usually you would keep much smaller amounts of data in transient (up to couple of MB).Use this method only for data (or operations) that do not change often, and set the expiration time to match the cycle of data changes.In effect, you are using it to render results that are generated through a series of database queries and storing the resulting HTML in the cache.The name of the transient record may not be longer than 45 characters, or 40 characters for “site transients” (used with multi-sites to store data at the network level).It works with WordPress since version 3.0.

Based on our SQL query, we can create a widget that relies on both caching methods. These are two approaches to the problem, and the two widgets will produce essentially the same output, but using different methods for data retrieval and results caching. As the administrator, you can set a title for the widget and the number of days to keep the results in the cache.

Both versions are simple and can be improved further (such as by selecting the post’s type or by formatting the output), but for this article they are enough.

The “raw” widget version stores an object with the SQL query results in the transient cache. In this case, the SQL query would return all columns from the wp_posts table and some columns from the wp_users table, along with information about the authors. Every time the widget loads, each post from our results set would get stored in the non-persistent cache object in the standard posts group, which is the same one used to store posts for normal WordPress operations. Because of this, functions such as get_permalink() can use the cached object to generate a URL to post. Information about the authors from the wp_users table is used to generate the URL for the archive of authors’ posts.

This widget is located in the method_raw.php file in the d4p_sa_method_raw class. The function get_data() is the most important part of the widget. It attempts to get data from the transient cache (on line 52). If that fails, get_data_real() is called to run the SQL query and return the data. This data is now saved into the transient cache (line 56). After we have the data, we store each post from the set into the non-persistent cache. The render function is simple; it displays the results as an unordered list.

The previous method works well, but it could have one problem. What if your permalink depends on categories (or other taxonomies) or you are running a query for a post type in a hierarchy? If that is the case, then generating a permalink for each post would require additional SQL queries. For example, to display 20 posts, you might need another 20 or more SQL queries. To fix the problem, we’ll change how we get the data and what is stored in the transient cache.

The second widget is located in the method_rendered.php file in the d4p_sa_method_rendered class. Within, the names of class methods are the same, so you can easily see now the difference between the two widgets. In this case, the transient cache is used in the render() method. We’re checking for cached data, and if that fails we use get_data() to get the data set and generate a rendered list of results. Now, we are caching the rendered HTML output! No matter how many extra SQL queries are needed to generate the HTML (for permalinks or whatever else you might need in the widget), they are run only once, and the complete HTML is cached. Until the cache expires, we are always displaying HTML rendered without the need for any additional SQL queries or processing.

You can download this D4P Smashing Authors plugin, which contains both widgets.

As you can see, implementing one or both caching methods is easy and could significantly improve the performance of your plugin. If a user of your plugin decides to use a specialized caching plugin, all the better, but make sure that your code is optimized.

(al)

Milan Petrovi?

Founder of Dev4Press, dedicated to development for WordPress, focusing on premium plugins. Dev4Press website offers wide selection of practical tutorials for WordPress. Milan is author of many popular WordPress plugins, including GD Star Rating, GD Press Tools and GD CPT Tools. Milan also developed several plugins for bbPress for WordPress powered forums: GD bbPress Toolbox.

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

Read more >>

Do-It-Yourself Caching Methods With WordPress

WordPress implements two different caching methods:

For this example, we will a create function named d4p_get_all_post_meta(), which will retrieve all meta data associated with a post. This first version doesn’t involve caching.

Every time you call this function for the same post ID, an SQL query will be executed. Here is the modified function that uses WordPress’ non-persistent cache:

As you can see, we’ve made just a small change to the existing code but potentially saved a lot of repeated database calls during the page’s loading.

function d4p_get_query_results() { global $wpdb; $data = $wpdb->get_results(' // SQL query // '); return $data;}

And here is the function using the transient cache, with a few extra lines to check whether the data is cached.

Both versions are simple and can be improved further (such as by selecting the post’s type or by formatting the output), but for this article they are enough.

You can download this D4P Smashing Authors plugin, which contains both widgets.

(al)

Milan Petrovi?

Read more >>

Replicating MySQL AES Encryption Methods With PHP

At our company, we process a lot of requests on the leading gift cards and coupons websites in the world. The senior developers had a meeting in late October to discuss working on a solution to replicate the MySQL functions of AES_ENCRYPT and AES_DECRYPT in the language of PHP. This article centers on what was produced by senior developer Derek Woods and how to use it in your own applications.

Security should be at the top of every developer’s mind when building an application that could hold sensitive data. We wanted to replicate MySQL’s functions because a lot of our data is already AES-encrypted in our database, and if you’re like us, you probably have that as well.

We will begin by examining why security and encryption matter at all levels of an application. Every application, no matter how large or small, needs some form of security and encryption. Any user data you have is extremely valuable to potential hackers and should be protected. Basic encryption should be used when your application stores a password or some other form of identifying information.

Different levels of sensitive data require different encryption algorithms. Knowing which level to use can be determined by answering the basic question, “Will I ever need access to the original data after it has been encrypted?” When storing a user’s password, using a heavily salted MD5 hash and storing that in the database is sufficient; then, you would use the same MD5 hashing on the input and compare the result from the database. When storing other sensitive data that will need to be returned to its original input, you would not be able to use a one-way hashing algorithm such as MD5; you would need a two-way encryption scheme to ensure that the original data can be returned after it’s been encrypted.

Encryption is only as powerful as the key used to protect it. Imagine that an attacker breaches your firewall and has an exact clone of your database; without the key, breaking the encryption that protects the sensitive data would be nearly impossible. Keeping the key in a safe place should always be priority number one. Many people use an ini file that’s read at runtime and that is not publicly accessible within the scope of the Web server. If your application requires two-way encryption, there are industry standards for protecting such data, one being AES encryption.

AES, which stands for Advanced Encryption Standard, was developed by Joan Daemen and Vincent Rijmen. They named their cipher, Rijndael, after a play on their two names. AES was announced as the winner of a five-year competition conducted by the National Institute of Standards and Technology (NIST) on 26 November 2001.

AES is a two-way encryption and decryption mechanism that provides a layer of security for sensitive data while still allowing the original data to be retrieved. To do this, it uses an encryption key that is used as a seed in the AES algorithm. As long as the key remains the same, the original data can be decrypted. This is necessary if your sensitive data needs to be returned to its original state.

AES uses a complex mathematical algorithm, which we will explore later, to combine two main concepts: confusion and diffusion. Confusion is a process that hides the relationship between the original data and the encrypted result. A classic example of this is the Caesar Cipher, which applies a simple shift of letters so that A becomes C, B becomes D, etc. Diffusion is a process that shifts, adjusts or otherwise alters the data in complex ways. This can be done using bit-shifts, replacements, additions, matrix manipulations and more. A combination of these two methods provides the layer of security that AES needs in order to give us a secure algorithm for our data.

In order to be bi-directional, the confusion and diffusion process is managed by the secret key that we provide to AES. Running the algorithm in one direction with a key and data will output an obfuscated string, thus encrypting the data. By passing that string back into the algorithm with the same key, running the algorithm in reverse will output the original data. Instead of attempting to keep the algorithm secret, the AES algorithm relies on complete key secrecy. According to its cryptographic storage guidelines, OWASP recommends rotating the key every one to three years. The guidelines also go through methods of rotating AES keys.

PHP provides AES implementation through the Mcrypt extension, which gives us a number of other ciphers as well. In addition to the algorithm itself, Mcrypt provides multiple modes that alter the security level of the AES algorithm to make it more secure. The two definitions that we will need to use in our PHP functions are MCRYPT_RIJNDAEL_256 and MCRYPT_MODE_ECB. Rijndael 256 is the encryption cipher that we will use for our AES encryption. Additionally, the code uses an “Electronic Code Book” that segments the data into blocks and encrypts them separately. Alternative and more secure modes are available, but MySQL uses ECB by default, so we will be crafting our PHP implementation around that.

Using MySQL’s AES encryption and decryption functions is very simple. Setting up requires less work, and it is generally far easier to understand. Apart from the obvious benefit of simplicity, there are three main reasons why PHP’s Mcrypt is superior to MySQL’s AES functions:

MySQL needs a database link between the application and database in order for encryption and decryption to occur. This could lead to unnecessary scalability issues and fatal errors if the database has internal failures, thus rendering your application unusable.PHP can do the same MySQL decryption and encryption with some effort but without a database connection, which improves the application’s speed and efficiency.MySQL often logs transactions, so if the database’s server has been compromised, then the log file would produce both the encryption key and the original value.

Out of the box, PHP’s Mcrypt functions unfortunately do not provide encrypted strings that match those of MySQL. There are a few reasons for this, but the root cause of the difference is the way that MySQL treats both the key and the input. To understand the causes, we have to delve into MySQL’s default AES encryption algorithm. The code segments presented below have been tested up to the latest version of MySQL, but MySQL could possibly alter their encryption scheme. The first problem we encounter is that MySQL will break up our secret key into 16-byte blocks and XOR the characters together from left to right. XOR is an exclusive disjunction, and if the two bytes are the same, then the output is 0; if they are different, then the output is 1. Additionally, it begins with a 16-byte block of null characters, so if we pass in a key of fewer than 16 bytes, then the rest of the key will contain null characters.

Say we have a 34-character key: 123456789abcdefghijklmnopqrstuvwxy. MySQL will start with the first 16 characters.

new_key = 123456789abcdefg

The second step taken is to XOR (?) the next 16 characters in the value in new_key with the first 16.

new_key = new_key ^ hijklmnopqrstuvwnew_key = 123456789abcdefg ^ hijklmnopqrstuvwnew_key = Y[Y_Y[YWI

Finally, the two remaining characters will be XOR’d starting from the left.

new_key = new_key ^ xynew_key = Y[Y_Y[YWI ^ xynew_key = !"Y_Y[YWI

This is, of course, drastically different from our original key, so if this process does not take place, then the return value from the decrypt/encrypt functions will be incorrect.

The second major difference between Mcrypt PHP and MySQL is that the length of the Mcrypt value must have padding to ensure it is a multiple of 16 bytes. There are numerous ways to do this, but the standard is to pad the key with the byte value equal to the number of bytes left over. So, if our value is 34 characters, then we would pad with a byte value of 14.

To calculate this, we use the following formula:

$pad_value = 16-(strlen($value) % 16);

Using our 34-character example, we would end up with this:

$pad_value = 16-(strlen("123456789abcdefghijklmnopqrstuvwxy") % 16);$pad_value = 16-(34 % 16);$pad_value = 16-(2);$pad_value = 14;

In the previous section, we dove into the issues surrounding the MySQL key used to encrypt and decrypt. Below is the function that’s used in both our encryption and decryption functions.

function mysql_aes_key($key){$new_key = str_repeat(chr(0), 16);for($i=0,$len=strlen($key);$i<$len;$i++){$new_key[$i%16] = $new_key[$i%16] ^ $key[$i];}return $new_key;}

First, we instantiate our key value with 16 null characters. Then we iterate through each character in the key and XOR it with the current position in new_key. Since we’re moving from left to right and using modulus 16, we will always be XOR’ing the correct characters together. This function changes our secret key to the MySQL standard and is the first step in achieving interoperability between PHP and MySQL.

We run into the last caveat when we pull the data from MySQL. For the data encrypted with AES, the padded values that we added before encryption will remain tacked onto the end after we decrypt. In general, this would go unnoticed if we were only fetching the data in order to display it; but if we’re using any of basic string functions on the decrypted data, such as strlen, then the results will be incorrect. There are a couple ways to handle this, and we will be removing all characters with a byte position of 0 through 16 from the right of our value since they are the only characters used in our padding algorithm.

The code below will handle the transformation.

$decrypted_value = rtrim($decrypted_value, "\0..\16");

Throwing all the concepts together, we have a few main points:

We have to transform our AES key to MySQL’s standards;We have to pad the value that we want to encrypt if it’s not a multiple of 16 bytes in size;We have to strip off the padded values after we decrypt the encrypted value from MySQL.

It’s advantageous to segment these concepts into components that can be reused throughout the project and in other areas that use AES encryption. The following two functions are the end result and perform the encryption and decryption before sending the data to MySQL for storage.

function aes_encrypt($val){$key = mysql_aes_key('Ralf_S_Engelschall__trainofthoughts');$pad_value = 16-(strlen($val) % 16);$val = str_pad($val, (16*(floor(strlen($val) / 16)+1)), chr($pad_value));return mcrypt_encrypt(MCRYPT_RIJNDAEL_128, $key, $val, MCRYPT_MODE_ECB, mcrypt_create_iv( mcrypt_get_iv_size(MCRYPT_RIJNDAEL_128, MCRYPT_MODE_ECB), MCRYPT_DEV_URANDOM));}

The first line is where we get the MySQL encoded key using the previously defined mysql_aes_key function. We are not using our real key in this article, but are instead paying homage to one of the creators of OpenSSL (among other technologies that he’s been involved in). The next line determines the character value with which to pad our data. The last two lines perform the actual padding of our data and call the mcrypt_encrypt function with the appropriate key and value. The return of this function will be the encrypted value that can be sent to MySQL for storage — or used anywhere else that requires encrypted data.

function aes_decrypt($val){$key = mysql_aes_key('Ralf_S_Engelschall__trainofthoughts');$val = mcrypt_decrypt(MCRYPT_RIJNDAEL_128, $key, $val, MCRYPT_MODE_ECB, mcrypt_create_iv( mcrypt_get_iv_size(MCRYPT_RIJNDAEL_128, MCRYPT_MODE_ECB), MCRYPT_DEV_URANDOM));return rtrim($val, "\0..\16");}

The first line of the decrypt function again generates the MySQL version of our secret key using mysql_aes_key. We then pass that key, along with the encrypted data, to the mcrypt_decrypt function. The final line returns the original data after stripping away any padded characters that we might have used in the encryption process.

To show that the encryption and decryption schemes here do in fact work, we must exercise both encryption and decryption functions in PHP and MySQL and compare the results. In this example, we have integrated the aes_encrypt/aes_decrypt and key function into a CakePHP model, and we are using Cake to run the database queries for MySQL. You can replace the CakePHP functions with mysql_query to obtain the results outside of Cake. In the first group, we are encoding the same data with the same key in both PHP and MySQL. We then base64_encode the result and print the data. The second group runs the MySQL encrypted data through PHP decrypt, and vice versa for the PHP encrypted data. We’re also outputting the result. The final block guarantees that the inputs and outputs are identical.

define('MY_KEY','Ralf_S_Engelschall__trainofthoughts');// Group 1$a = $this->User->aes_encrypt('test');echo base64_encode($a).'';$result = $this->User->query("SELECT AES_ENCRYPT('test', '".MY_KEY."') AS enc");$b = $result[0][0]['enc'];echo base64_encode($b).'';// Group 2$result = $this->User->query("SELECT AES_DECRYPT('".$a."', '".MY_KEY."') AS decc");$c = $result[0][0]['decc'];echo $c."
";$d = $this->User->aes_decrypt($b);echo $d."
";// Comparisonvar_dump($a===$b);var_dump($c===$d);

The snippet below is the output when you run the PHP commands listed above.

L8534Dj1sH6IRFrUXXBkkA==L8534Dj1sH6IRFrUXXBkkA==testtestbool(true) bool(true)

AES encryption can be a bit painful if you are not familiar with the specifics of the algorithm or familiar with any differences between implementations that you might have to work with in your various libraries and software packages. As you can see, it is indeed possible to use native PHP functions to handle encryption and decryption and to actually make it work seamlessly with any legacy MySQL-only encrypted data that your application might have. These methods should be used regardless so that you can use MySQL to decrypt data if a scenario arises in which that is the only option.

(al)

Chad Smith & Derek Woods

Chad Smith is a Senior Programmer for OmniPrepaid LLC, which operates the largest gift cards, and coupons sites in the world. He is based in Pittsburgh, PA and is passionate about UX, API, jQuery, CSS, PHP and Symfony2/Doctrine development. In his spare time he enjoys spending time with his two children, wife, and enjoys watching The Easy API grow. Derek Woods is a Senior Developer at OmniPrepaid LLC based in Pittsburgh, PA. He spends his time creating products across a wide range of technologies from Mongo and MySQL to PHP, Node.js and Java. Every now and then he blogs about it.

Read more >>