Powerful Command Line Tools For Developers

Life as a Web developer can be hard when things start going wrong. The problem could be in any number of places. Is there a problem with the request you’re sending, is the problem with the response, is there a problem with a request in a third party library you’re using, is an external API failing?

Good tools are invaluable in figuring out where problems lie, and can also help to prevent problems from occurring in the first place, or just help you to be more efficient in general. Command line tools are particularly useful because they lend themselves well to automation and scripting, where they can be combined and reused in all sorts of different ways. Here we cover six particularly powerful and versatile tools which can help make your life a little bit easier.

(Image credit: kolnikcollection)

Curl is a network transfer tool that’s very similar to Wget, the main difference being that by default Wget saves to a file, and curl outputs to the command line. This makes it really simple to see the contents of a website. Here, for example, we can get our current IP from the ifconfig.me website:

$ curl ifconfig.me
93.96.141.93

Curl’s -i (show headers) and -I (show only headers) options make it a great tool for debugging HTTP responses and finding out exactly what a server is sending to you:

$ curl -I news.ycombinator.comHTTP/1.1 200 OKContent-Type: text/html; charset=utf-8Cache-Control: privateConnection: close

The -L option is handy, and makes curl automatically follow redirects. Curl has support for HTTP Basic authentication, cookies, manually setting headers and much, much more.

For serious network packet analysis there’s Wireshark, with its thousands of settings, filters and configuration options. There’s also a command-line version, TShark. For simple tasks I find Wireshark can be overkill, so unless I need something more powerful, ngrep is my tool of choice. It allows you to do with network packets what grep does with files.

For Web traffic you almost always want the -W byline option, which preserves linebreaks, and -q is a useful argument which suppresses some additional output about non-matching packets. Here’s an example that captures all packets that contain GET or POST:

ngrep -q -W byline "^(GET|POST) .*"

You can also pass in additional packet filter options, such as limiting the matched packets to a certain host, IP or port. Here we filter all traffic going to or coming from Google, using port 80 and containing the term “search.”

ngrep -q -W byline "search" host www.google.com and port 80

Netcat, or nc, is a self-described networking Swiss Army knife. It’s a very simple but also very powerful and versatile application that allows you to create arbitrary network connections. Here we see it being used as a port scanner:

$ nc -z example.com 20-100Connection to example.com 22 port [tcp/ssh] succeeded!Connection to example.com 80 port [tcp/http] succeeded!

In addition to creating arbitrary connections, Netcat can also listen for incoming connections. Here we use this feature of nc, combined with tar, to very quickly and efficiently copy files between servers. On the server, run:

$ nc -l 9090 | tar -xzf -

And on the client:

$ tar -czf dir/ | nc server 9090

We can use Netcat to expose any application over the network. Here we expose a shell over port 8080:

$ mkfifo backpipe$ nc -l 8080 0 backpipe

We can now access the server from any client:

$ nc example.com 8080uname -aLinux li228-162 2.6.39.1-linode34 ##1 SMP Tue Jun 21 10:29:24 EDT 2011 i686 GNU/Linux

While the last two examples are slightly contrived (in reality you’d be more likely to use tools such as rsync to copy files and SSH to remotely access a server), they do show the power and flexibility of Netcat, and hint at all of the different things you can achieve by combining Netcat with other applications.

Sshuttle allows you to securely tunnel your traffic via any server you have SSH access to. It’s extremely easy to set up and use, not requiring you to install any software on the server or change any local proxy settings.

By tunneling your traffic over SSH, you secure yourself against tools like Firesheep and dsniff when you’re on unsecured public Wi-Fi or other untrusted networks. All network communication, including DNS requests, can be sent via your SSH server:

$ sshuttle -r --dns 0/0

If you provide the --daemon argument, sshuttle will run in the background as a daemon. Combined with some other options, you can make aliases to simply and quickly start and stop tunneling your traffic:

alias tunnel='sshuttle --D --pidfile=/tmp/sshuttle.pid -r --dns 0/0'alias stoptunnel='[[ -f /tmp/sshuttle.pid ]] && kill `cat /tmp/sshuttle.pid`'

You can also use sshuttle to get around the IP-based geolocation filters that are now used by many services, such as BBC’s iPlayer, which requires you to be in the UK, and Turntable, which requires you to be in the US. To do this, you’ll need access to a server in the target country. Amazon has a free tier of EC2 Micro instances that are available in many countries, or you can find a cheap virtual private server (VPS) in almost any country in the world.

In this scenario, rather than tunneling all of our traffic, we might want to send only traffic for the service we are targeting. Unfortunately, sshuttle only accepts IP address arguments and not hostnames, so we need to make use of dig to first resolve the hostname:

$ sshuttle -r `dig +short `

Siege is a HTTP benchmarking tool. In addition to load-testing features, it has a handy -g option that is very similar to curl’s -iL, except it also shows you the request headers. Here’s an example with Google (I’ve removed some headers for brevity):

$ siege -g www.google.comGET / HTTP/1.1Host: www.google.comUser-Agent: JoeDog/1.00 [en] (X11; I; Siege 2.70)Connection: closeHTTP/1.1 302 FoundLocation: http://www.google.co.uk/Content-Type: text/html; charset=UTF-8Server: gwsContent-Length: 221Connection: closeGET / HTTP/1.1Host: www.google.co.ukUser-Agent: JoeDog/1.00 [en] (X11; I; Siege 2.70)Connection: closeHTTP/1.1 200 OKContent-Type: text/html; charset=ISO-8859-1X-XSS-Protection: 1; mode=blockConnection: close

What Siege is really great at is server load testing. Just like ab (an Apache HTTP server benchmarking tool), you can send a number of concurrent requests to a site, and see how it handles the traffic. With the following command, we’ll test Google with 20 concurrent connections for 30 seconds, and then get a nice report at the end:

$ siege -c20 www.google.co.uk -b -t30s...Lifting the server siege... done.Transactions: 1400 hitsAvailability: 100.00 %Elapsed time: 29.22 secsData transferred: 13.32 MBResponse time: 0.41 secsTransaction rate: 47.91 trans/secThroughput: 0.46 MB/secConcurrency: 19.53Successful transactions: 1400Failed transactions: 0Longest transaction: 4.08Shortest transaction: 0.08

One of the most useful features of Siege is that it can take a file of URLs as an input, and then hit those URLs rather than just a single page. This is great for load testing, because you can replay real traffic against your site and see how it performs, rather than just hitting the same URL again and again. Here’s how you would use Siege to replay your Apache logs against another server to load test it:

$ cut -d ' ' -f7 /var/log/apache2/access.log > urls.txt$ siege -c -b -f urls.txt

Mitmproxy is an SSL-capable, man-in-the-middle HTTP proxy that allows you to inspect both HTTP and HTTPS traffic, and rewrite requests on the fly. The application has been behind quite a few iOS application privacy scandals, including Path’s address book upload scandal. Its ability to rewrite requests on the fly has also been used to target iOS, including setting a fake high score in GameCenter.

Far from only being useful to see what mobile applications are sending over the wire or for faking high scores, mitmproxy can help out with a whole range of Web development tasks. For example, instead of constantly hitting F5 or clearing your cache to make sure you’re seeing the latest content, you can run

$ mitmproxy --anticache

which will automatically strip all cache-control headers and make sure you always get fresh content. Unfortunately it doesn’t automatically set up forwarding for you like sshuttle does, so after starting mitmproxy you still need to change your system-wide or browser-specific proxy settings.

Another extremely handy feature of mitmproxy is the ability to record and replay HTTP interactions. The official documentation gives an example of a wireless network login. The same technique can be used as a basic Web testing framework. For example, to confirm that your user signup flow works, you can start recording the session:

$ mitmdump -w user-signup

Then go through the user signup process, which at this point should work as expected. Stop recording the session with Ctrl + C. At any point we can then replay what was recorded and check for the 200 status code:

$ mitmdump -c user-signup | tail -n1 | grep 200 && echo "OK" || echo "FAIL"

If the signup flow gets broken at any point, we’ll see a FAIL message, rather than an OK. You could create a whole suite of these tests and run them regularly to make sure you get notified if you ever accidentally break anything on your site.

(cp)

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

Read more >>

Tweet

All the Tools a Tech Needs

by Alison - on August 15th, 2012

iFixit was founded in 2003 by two college students. During a quest to repair a laptop they tinkered around enough to realize the huge opportunity to share easy-to-use repair instructions. Years later, iFixit has become one of the best device repair resources on the Internet. Their library of manuals repair guides numbers into the thousands covering everything from computers to phones to gaming devices to small appliances. And most content is contributed via their community of experts. Visit iFixit if interested in contributing your knowledge!

Ever tried to repair a gadget? Inevitably – even if you have a manual – you don’t quite have the requisite tools to hack into your broken gizmo. iFixit sees this need and has also become a resource for parts and tools needed to complete many major repairs.  For the truly adventurous repair-it-yourself person, the iFixit ProTech Toolkit provides over 70 tools at your fingertips. Included are 54 bits, spudgers, opening tools, tweezers, anti-static wrist strap and even a suction cup all for just $59.95.  Expansion kits exist if bigger jobs are in your future. And a smaller home kit exists for those that are a little intimidated by – or may not have the need for – as many tools. Check out all iFixit kits. And BEFORE you open your electronics and attempt repairs ensure that your instruments are up to the task.

There are various iFixit kits available on Amazon.com.

Read more >>

Tweet

JavaScript Profiling With The Chrome Developer Tools

Your website works. Now let’s make it work faster. Website performance is about two things: how fast the page loads, and how fast the code on it runs. Plenty of services make your website load faster, from minimizers to CDNs, but making it run faster is up to you.

Little changes in your code can have gigantic performance impacts. A few lines here or there could mean the difference between a blazingly fast website and the dreaded “Unresponsive Script” dialog. This article shows you a few ways to find those lines of code with Chrome Developer Tools.

We’ll look at a simple application called a color sorter, which presents a grid of rainbow colors that you can drag and drop to mix up. Each dot is a div tag with a little CSS to make it look like a circle.

Generating my rainbow colors was a little tricky, so I got help from “Making Annoying Rainbows in JavaScript.”

The page loads pretty fast, but it still takes a moment and blinks a little before it paints. Time to profile the page and make it faster.

Always start performance-improvement projects with a baseline understanding of how fast or slow your application already is. The baseline will let you know whether you’re making improvements and help you make tradeoffs. For this article, we’ll use Chrome Developer Tools.

The profiler is part of Chrome Developer Tools, which is always available in Chrome. Click the “Tools” menu under the little wrench to open it. Firebug has some profiling tools, too, but the WebKit browsers (Chrome and Safari) are best at profiling code and showing timelines. Chrome also offers an excellent tool for event tracing, called Speed Tracer.

To establish our baseline, we’ll start recording in the “Timeline” tab, load our page and then stop the recording. (To start recording once Chrome Developer Tools is open, click the “Timeline” tab, and then the small black circle icon for “Record” at the very bottom of the window.) Chrome is smart about not starting to record until the page starts to load. I run it three times and take the average, in case my computer runs slowly during the first test.

My average baseline — i.e. the time between the first request for the page and the final painting of the page in the browser — is 1.25 seconds. That’s not bad, but it’s not great for such a small page.

I want to make my code run faster, but I’m not sure what’s making it slow. The profiler helps me find out.

The timeline tells us how long our code took to run, but that doesn’t help us know what’s going on while it’s running. We could make changes and run the timeline again and again, but that’s just shooting in the dark. The “Profiles” tab gives us a better way to see what’s going on.

Profilers show us which functions take the most time. Let’s make our baseline profile by switching to the “Profiles” tab in Chrome Developer Tools, where three types of profiling are offered:

JavaScript CPU profile
Shows how much CPU time our JavaScript is taking.CSS selector profile
Shows how much CPU time is spent processing CSS selectors.Heap snapshot
Shows how memory is being used by our JavaScript objects.

We want to make our JavaScript run faster, so we’ll use the CPU profiling. We start the profile, refresh the page and then stop the profiler.

The first thing that’s clear from the profile is that a lot is going on. The color sorter uses jQuery and jQuery UI, which are doing a lot of stuff like managing plugins and parsing regular expressions. I can also see that two of my functions are at the top of the list: decimalToHex and makeColorSorter. These two functions take a total of 13.2% of my CPU time, so they’re a good place to start making improvements.

We can click the arrow next to the function calls to open the complete function-call stack. Expanding them, I see that decimalToHex is called from makeColorSorter, and makeColorSorter is called from $(document).ready.

Here’s the code:

$(document).ready(function() { makeColorSorter(.05, .05, .05, 0, 2, 4, 128, 127, 121); makeSortable();});

Knowing where they’re called from also makes clear that making the colors sortable isn’t my biggest performance problem. Performance issues resulting from the addition of a lot of sortables is common, but my code is taking more time to add DOM elements than to make them sortable.

I want to start making those functions faster, but first I want to isolate my changes. A lot happens when the page loads, and I want to get all of that out of my profile.

Instead of loading the color sorter when the document is ready, I’ll make a second version that waits until I press a button to load the color sorter. This isolates it from the document loading and helps me profile just the code. I can change it back once I’m done tuning performance.

Let’s call the new function testColorSorter and bind it to a clickable button:

function testColorSorter() { makeColorSorter(.05, .05, .05, 0, 2, 4, 128, 127, 121); makeSortable();}

Changing the application before we profile could alter the performance of the application unexpectedly. This change looks pretty safe, but I still want to run the profiler again to see whether I’ve accidentally changed anything else. I’ll create this new profile by starting the profiler, pressing the button in the app and then stopping the profile.

The first thing to notice is that the decimalToHex function is now taking up 4.23% of the time to load; it’s what the code spends the most time on. Let’s create a new baseline to see how much the code improves in this scenario.

A few events occur before I press the button, but I only care about how long it took between the times the mouse was clicked and the browser painted the color sorter. The mouse button was clicked at 390 milliseconds, and the paint event happened at 726 milliseconds; 726 minus 390 equals my baseline of 336 milliseconds. Just as with the first baseline, I ran it three times and took the average time.

At this point, I know where to look and how long the code takes to run. Now we’re ready to start fixing the problem.

The profiler only tells us which function is causing the problem, so we need to look into it and understand what it does.

function decimalToHex(d) { var hex = Number(d).toString(16); hex = "00".substr(0, 2 - hex.length) + hex; console.log('converting ' + d + ' to ' + hex); return hex;}

Each dot in the color sorter takes a background color value in hex format, such as #86F01B or #2456FE. These values represent the red, green and blue values of the color. For example, has a background color of #2456FE, which means a red value of 36, a green value of 86 and a blue value of 254. Each value must be between 0 and 255.

The decimalToHex function converts these RGB colors to hex colors that we can use on the page.

The function is pretty simple, but I’ve left a console.log message in there, which is just some debugging code we can remove.

The decimalToHex function also adds padding to the beginning of the number. This is important because some base-10 numbers result in a single hex digit; 12 in base 10 is C in hex, but CSS requires two digits. We can make the conversion faster by making it a little less generic. I know that the numbers to be padded each have one digit, so we can rewrite the function like this:

function decimalToHex(d) { var hex = Number(d).toString(16); return hex.length === 1 ? '0' + hex : hex; }

Version three of the color sorter changes the string only when it needs the padding and doesn’t have to call substr. With this new function, our runtime is 137 milliseconds. By profiling the code again, I can see that the decimalToHex function now takes only 0.04% of the total time — putting it way down the list.

We can also see that the function using the most CPU is e.extend.merge from jQuery. I’m not sure what that function does because the code is minimized. I could add the development version of jQuery, but I can see that the function is getting called from makeColorSorter, so let’s make that one faster next.

The rainbow colors in the color sorter are generated from a sine wave. The code looks at a center point in the color spectrum and creates a wave through that center point over a specified width. This changes the colors into a rainbow pattern. We can also change the colors in the rainbow by changing the frequency of the red, green and blue.

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); console.log('red: ' + decimalToHex(red)); console.log('green: ' + decimalToHex(green)); console.log('blue: ' + decimalToHex(blue)); var div = $('

'); div.css('background-color', '#' + decimalToHex(red) + decimalToHex(green) + decimalToHex(blue)); $('#colors').append(div); }}

We could take out more console.log functions. The calls are especially bad because each is also calling the decimalToHex function, which means that decimalToHex is effectively being called twice as often as it should.

This function changes the DOM a lot. Every time the loop runs, it adds a new div to the colors div tag. This makes me wonder whether that’s what the e.extend.merge function was doing. The profiler makes it easy to tell with a simple experiment.

Instead of adding a new div each time the loop runs, I want to add all of the div tags at once. Let’s create a variable to hold them, and then add them once at the end.

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { var colors = ""; for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); colors += '

'; } $('#colors').append(colors);}

This small change in the code means that the DOM changes once, when it adds all of the div tags. Testing that with the timeline, we see that the runtime between the click and the paint events is now 31 milliseconds. This one DOM change has brought the time for version four down by about 87%. We can also run the profiler again and see that the e.extend.merge function now takes up such a small percentage of the time that it doesn’t show up on the list.

We could make the code one notch faster by removing the decimalToHex function entirely. CSS supports RGB colors, so we don’t need to convert them to hex. Now we can write our makeColorSorter function like this:

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { var colors = ""; for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); colors += '

'; } $('#colors').append(colors);}

Version five runs in only 26 milliseconds and uses 18 lines of code for what used to take 28 lines.

Real-world applications are much more complex than this color sorter, but profiling them follows the same basic steps:

Establish a baseline so that you know where you’re starting from.Isolate the problem from any other code running in the application.Make it faster in a controlled environment, with frequent timelines and profiles.

There are a few other rules to follow when tuning performance:

Start with the slowest parts first so that you get the most improvement for the time spent tuning.Control the environment. If you switch computers or make any other major changes, always run a new baseline.Repeat the analysis to prevent anomalies on your computer from skewing the results.

Everyone wants their website to run faster. You have to develop new features, but new features usually make a website run slower. So, investing time in tuning the performance does pay off.

Profiling and tuning cut the final color sorter’s runtime by over 92%. How much faster could your website be?

(al) (km)

Zack Grossbart

Zack Grossbart is an engineer, designer, and author. He's a founding member of the Spiffy UI project, the architect of the WordPress Editorial Calendar, and a Consulting Engineer with NetIQ. Zack began loading DOS from a floppy disk when he was five years old. He first worked professionally with computers when he was 15 and started his first software company when he was 16. Zack lives in Cambridge, Massachusetts with his wife and daughter.

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

Read more >>

Tweet

JavaScript Profiling With The Chrome Developer Tools

Your website works. Now let’s make it work faster. Website performance is about two things: how fast the page loads, and how fast the code on it runs. Plenty of services make your website load faster, from minimizers to CDNs, but making it run faster is up to you.

Little changes in your code can have gigantic performance impacts. A few lines here or there could mean the difference between a blazingly fast website and the dreaded “Unresponsive Script” dialog. This article shows you a few ways to find those lines of code with Chrome Developer Tools.

We’ll look at a simple application called a color sorter, which presents a grid of rainbow colors that you can drag and drop to mix up. Each dot is a div tag with a little CSS to make it look like a circle.

Generating my rainbow colors was a little tricky, so I got help from “Making Annoying Rainbows in JavaScript.”

The page loads pretty fast, but it still takes a moment and blinks a little before it paints. Time to profile the page and make it faster.

Always start performance-improvement projects with a baseline understanding of how fast or slow your application already is. The baseline will let you know whether you’re making improvements and help you make tradeoffs. For this article, we’ll use Chrome Developer Tools.

The profiler is part of Chrome Developer Tools, which is always available in Chrome. Click the “Tools” menu under the little wrench to open it. Firebug has some profiling tools, too, but the WebKit browsers (Chrome and Safari) are best at profiling code and showing timelines. Chrome also offers an excellent tool for event tracing, called Speed Tracer.

To establish our baseline, we’ll start recording in the “Timeline” tab, load our page and then stop the recording. (To start recording once Chrome Developer Tools is open, click the “Timeline” tab, and then the small black circle icon for “Record” at the very bottom of the window.) Chrome is smart about not starting to record until the page starts to load. I run it three times and take the average, in case my computer runs slowly during the first test.

My average baseline — i.e. the time between the first request for the page and the final painting of the page in the browser — is 1.25 seconds. That’s not bad, but it’s not great for such a small page.

I want to make my code run faster, but I’m not sure what’s making it slow. The profiler helps me find out.

The timeline tells us how long our code took to run, but that doesn’t help us know what’s going on while it’s running. We could make changes and run the timeline again and again, but that’s just shooting in the dark. The “Profiles” tab gives us a better way to see what’s going on.

Profilers show us which functions take the most time. Let’s make our baseline profile by switching to the “Profiles” tab in Chrome Developer Tools, where three types of profiling are offered:

JavaScript CPU profile
Shows how much CPU time our JavaScript is taking.CSS selector profile
Shows how much CPU time is spent processing CSS selectors.Heap snapshot
Shows how memory is being used by our JavaScript objects.

We want to make our JavaScript run faster, so we’ll use the CPU profiling. We start the profile, refresh the page and then stop the profiler.

The first thing that’s clear from the profile is that a lot is going on. The color sorter uses jQuery and jQuery UI, which are doing a lot of stuff like managing plugins and parsing regular expressions. I can also see that two of my functions are at the top of the list: decimalToHex and makeColorSorter. These two functions take a total of 13.2% of my CPU time, so they’re a good place to start making improvements.

We can click the arrow next to the function calls to open the complete function-call stack. Expanding them, I see that decimalToHex is called from makeColorSorter, and makeColorSorter is called from $(document).ready.

Here’s the code:

$(document).ready(function() { makeColorSorter(.05, .05, .05, 0, 2, 4, 128, 127, 121); makeSortable();});

Knowing where they’re called from also makes clear that making the colors sortable isn’t my biggest performance problem. Performance issues resulting from the addition of a lot of sortables is common, but my code is taking more time to add DOM elements than to make them sortable.

I want to start making those functions faster, but first I want to isolate my changes. A lot happens when the page loads, and I want to get all of that out of my profile.

Instead of loading the color sorter when the document is ready, I’ll make a second version that waits until I press a button to load the color sorter. This isolates it from the document loading and helps me profile just the code. I can change it back once I’m done tuning performance.

Let’s call the new function testColorSorter and bind it to a clickable button:

function testColorSorter() { makeColorSorter(.05, .05, .05, 0, 2, 4, 128, 127, 121); makeSortable();}

Changing the application before we profile could alter the performance of the application unexpectedly. This change looks pretty safe, but I still want to run the profiler again to see whether I’ve accidentally changed anything else. I’ll create this new profile by starting the profiler, pressing the button in the app and then stopping the profile.

The first thing to notice is that the decimalToHex function is now taking up 4.23% of the time to load; it’s what the code spends the most time on. Let’s create a new baseline to see how much the code improves in this scenario.

A few events occur before I press the button, but I only care about how long it took between the times the mouse was clicked and the browser painted the color sorter. The mouse button was clicked at 390 milliseconds, and the paint event happened at 726 milliseconds; 726 minus 390 equals my baseline of 336 milliseconds. Just as with the first baseline, I ran it three times and took the average time.

At this point, I know where to look and how long the code takes to run. Now we’re ready to start fixing the problem.

The profiler only tells us which function is causing the problem, so we need to look into it and understand what it does.

function decimalToHex(d) { var hex = Number(d).toString(16); hex = "00".substr(0, 2 - hex.length) + hex; console.log('converting ' + d + ' to ' + hex); return hex;}

Each dot in the color sorter takes a background color value in hex format, such as #86F01B or #2456FE. These values represent the red, green and blue values of the color. For example, has a background color of #2456FE, which means a red value of 36, a green value of 86 and a blue value of 254. Each value must be between 0 and 255.

The decimalToHex function converts these RGB colors to hex colors that we can use on the page.

The function is pretty simple, but I’ve left a console.log message in there, which is just some debugging code we can remove.

The decimalToHex function also adds padding to the beginning of the number. This is important because some base-10 numbers result in a single hex digit; 12 in base 10 is C in hex, but CSS requires two digits. We can make the conversion faster by making it a little less generic. I know that the numbers to be padded each have one digit, so we can rewrite the function like this:

function decimalToHex(d) { var hex = Number(d).toString(16); return hex.length === 1 ? '0' + hex : hex; }

Version three of the color sorter changes the string only when it needs the padding and doesn’t have to call substr. With this new function, our runtime is 137 milliseconds. By profiling the code again, I can see that the decimalToHex function now takes only 0.04% of the total time — putting it way down the list.

We can also see that the function using the most CPU is e.extend.merge from jQuery. I’m not sure what that function does because the code is minimized. I could add the development version of jQuery, but I can see that the function is getting called from makeColorSorter, so let’s make that one faster next.

The rainbow colors in the color sorter are generated from a sine wave. The code looks at a center point in the color spectrum and creates a wave through that center point over a specified width. This changes the colors into a rainbow pattern. We can also change the colors in the rainbow by changing the frequency of the red, green and blue.

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); console.log('red: ' + decimalToHex(red)); console.log('green: ' + decimalToHex(green)); console.log('blue: ' + decimalToHex(blue)); var div = $('

'); div.css('background-color', '#' + decimalToHex(red) + decimalToHex(green) + decimalToHex(blue)); $('#colors').append(div); }}

We could take out more console.log functions. The calls are especially bad because each is also calling the decimalToHex function, which means that decimalToHex is effectively being called twice as often as it should.

This function changes the DOM a lot. Every time the loop runs, it adds a new div to the colors div tag. This makes me wonder whether that’s what the e.extend.merge function was doing. The profiler makes it easy to tell with a simple experiment.

Instead of adding a new div each time the loop runs, I want to add all of the div tags at once. Let’s create a variable to hold them, and then add them once at the end.

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { var colors = ""; for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); colors += '

'; } $('#colors').append(colors);}

This small change in the code means that the DOM changes once, when it adds all of the div tags. Testing that with the timeline, we see that the runtime between the click and the paint events is now 31 milliseconds. This one DOM change has brought the time for version four down by about 87%. We can also run the profiler again and see that the e.extend.merge function now takes up such a small percentage of the time that it doesn’t show up on the list.

We could make the code one notch faster by removing the decimalToHex function entirely. CSS supports RGB colors, so we don’t need to convert them to hex. Now we can write our makeColorSorter function like this:

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { var colors = ""; for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); colors += '

'; } $('#colors').append(colors);}

Version five runs in only 26 milliseconds and uses 18 lines of code for what used to take 28 lines.

Real-world applications are much more complex than this color sorter, but profiling them follows the same basic steps:

Establish a baseline so that you know where you’re starting from.Isolate the problem from any other code running in the application.Make it faster in a controlled environment, with frequent timelines and profiles.

There are a few other rules to follow when tuning performance:

Start with the slowest parts first so that you get the most improvement for the time spent tuning.Control the environment. If you switch computers or make any other major changes, always run a new baseline.Repeat the analysis to prevent anomalies on your computer from skewing the results.

Everyone wants their website to run faster. You have to develop new features, but new features usually make a website run slower. So, investing time in tuning the performance does pay off.

Profiling and tuning cut the final color sorter’s runtime by over 92%. How much faster could your website be?

(al) (km)

Zack Grossbart

Zack Grossbart is an engineer, designer, and author. He's a founding member of the Spiffy UI project, the architect of the WordPress Editorial Calendar, and a Consulting Engineer with NetIQ. Zack began loading DOS from a floppy disk when he was five years old. He first worked professionally with computers when he was 15 and started his first software company when he was 16. Zack lives in Cambridge, Massachusetts with his wife and daughter.

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

Read more >>

Tweet

JavaScript Profiling With The Chrome Developer Tools

Your website works. Now let’s make it work faster. Website performance is about two things: how fast the page loads, and how fast the code on it runs. Plenty of services make your website load faster, from minimizers to CDNs, but making it run faster is up to you.

Little changes in your code can have gigantic performance impacts. A few lines here or there could mean the difference between a blazingly fast website and the dreaded “Unresponsive Script” dialog. This article shows you a few ways to find those lines of code with Chrome Developer Tools.

We’ll look at a simple application called a color sorter, which presents a grid of rainbow colors that you can drag and drop to mix up. Each dot is a div tag with a little CSS to make it look like a circle.

Generating my rainbow colors was a little tricky, so I got help from “Making Annoying Rainbows in JavaScript.”

The page loads pretty fast, but it still takes a moment and blinks a little before it paints. Time to profile the page and make it faster.

Always start performance-improvement projects with a baseline understanding of how fast or slow your application already is. The baseline will let you know whether you’re making improvements and help you make tradeoffs. For this article, we’ll use Chrome Developer Tools.

The profiler is part of Chrome Developer Tools, which is always available in Chrome. Click the “Tools” menu under the little wrench to open it. Firebug has some profiling tools, too, but the WebKit browsers (Chrome and Safari) are best at profiling code and showing timelines. Chrome also offers an excellent tool for event tracing, called Speed Tracer.

To establish our baseline, we’ll start recording in the “Timeline” tab, load our page and then stop the recording. (To start recording once Chrome Developer Tools is open, click the “Timeline” tab, and then the small black circle icon for “Record” at the very bottom of the window.) Chrome is smart about not starting to record until the page starts to load. I run it three times and take the average, in case my computer runs slowly during the first test.

My average baseline — i.e. the time between the first request for the page and the final painting of the page in the browser — is 1.25 seconds. That’s not bad, but it’s not great for such a small page.

I want to make my code run faster, but I’m not sure what’s making it slow. The profiler helps me find out.

The timeline tells us how long our code took to run, but that doesn’t help us know what’s going on while it’s running. We could make changes and run the timeline again and again, but that’s just shooting in the dark. The “Profiles” tab gives us a better way to see what’s going on.

Profilers show us which functions take the most time. Let’s make our baseline profile by switching to the “Profiles” tab in Chrome Developer Tools, where three types of profiling are offered:

JavaScript CPU profile
Shows how much CPU time our JavaScript is taking.CSS selector profile
Shows how much CPU time is spent processing CSS selectors.Heap snapshot
Shows how memory is being used by our JavaScript objects.

We want to make our JavaScript run faster, so we’ll use the CPU profiling. We start the profile, refresh the page and then stop the profiler.

The first thing that’s clear from the profile is that a lot is going on. The color sorter uses jQuery and jQuery UI, which are doing a lot of stuff like managing plugins and parsing regular expressions. I can also see that two of my functions are at the top of the list: decimalToHex and makeColorSorter. These two functions take a total of 13.2% of my CPU time, so they’re a good place to start making improvements.

We can click the arrow next to the function calls to open the complete function-call stack. Expanding them, I see that decimalToHex is called from makeColorSorter, and makeColorSorter is called from $(document).ready.

Here’s the code:

$(document).ready(function() { makeColorSorter(.05, .05, .05, 0, 2, 4, 128, 127, 121); makeSortable();});

Knowing where they’re called from also makes clear that making the colors sortable isn’t my biggest performance problem. Performance issues resulting from the addition of a lot of sortables is common, but my code is taking more time to add DOM elements than to make them sortable.

I want to start making those functions faster, but first I want to isolate my changes. A lot happens when the page loads, and I want to get all of that out of my profile.

Instead of loading the color sorter when the document is ready, I’ll make a second version that waits until I press a button to load the color sorter. This isolates it from the document loading and helps me profile just the code. I can change it back once I’m done tuning performance.

Let’s call the new function testColorSorter and bind it to a clickable button:

function testColorSorter() { makeColorSorter(.05, .05, .05, 0, 2, 4, 128, 127, 121); makeSortable();}

Changing the application before we profile could alter the performance of the application unexpectedly. This change looks pretty safe, but I still want to run the profiler again to see whether I’ve accidentally changed anything else. I’ll create this new profile by starting the profiler, pressing the button in the app and then stopping the profile.

The first thing to notice is that the decimalToHex function is now taking up 4.23% of the time to load; it’s what the code spends the most time on. Let’s create a new baseline to see how much the code improves in this scenario.

A few events occur before I press the button, but I only care about how long it took between the times the mouse was clicked and the browser painted the color sorter. The mouse button was clicked at 390 milliseconds, and the paint event happened at 726 milliseconds; 726 minus 390 equals my baseline of 336 milliseconds. Just as with the first baseline, I ran it three times and took the average time.

At this point, I know where to look and how long the code takes to run. Now we’re ready to start fixing the problem.

The profiler only tells us which function is causing the problem, so we need to look into it and understand what it does.

function decimalToHex(d) { var hex = Number(d).toString(16); hex = "00".substr(0, 2 - hex.length) + hex; console.log('converting ' + d + ' to ' + hex); return hex;}

Each dot in the color sorter takes a background color value in hex format, such as #86F01B or #2456FE. These values represent the red, green and blue values of the color. For example, has a background color of #2456FE, which means a red value of 36, a green value of 86 and a blue value of 254. Each value must be between 0 and 255.

The decimalToHex function converts these RGB colors to hex colors that we can use on the page.

The function is pretty simple, but I’ve left a console.log message in there, which is just some debugging code we can remove.

The decimalToHex function also adds padding to the beginning of the number. This is important because some base-10 numbers result in a single hex digit; 12 in base 10 is C in hex, but CSS requires two digits. We can make the conversion faster by making it a little less generic. I know that the numbers to be padded each have one digit, so we can rewrite the function like this:

function decimalToHex(d) { var hex = Number(d).toString(16); return hex.length === 1 ? '0' + hex : hex; }

Version three of the color sorter changes the string only when it needs the padding and doesn’t have to call substr. With this new function, our runtime is 137 milliseconds. By profiling the code again, I can see that the decimalToHex function now takes only 0.04% of the total time — putting it way down the list.

We can also see that the function using the most CPU is e.extend.merge from jQuery. I’m not sure what that function does because the code is minimized. I could add the development version of jQuery, but I can see that the function is getting called from makeColorSorter, so let’s make that one faster next.

The rainbow colors in the color sorter are generated from a sine wave. The code looks at a center point in the color spectrum and creates a wave through that center point over a specified width. This changes the colors into a rainbow pattern. We can also change the colors in the rainbow by changing the frequency of the red, green and blue.

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); console.log('red: ' + decimalToHex(red)); console.log('green: ' + decimalToHex(green)); console.log('blue: ' + decimalToHex(blue)); var div = $('

'); div.css('background-color', '#' + decimalToHex(red) + decimalToHex(green) + decimalToHex(blue)); $('#colors').append(div); }}

We could take out more console.log functions. The calls are especially bad because each is also calling the decimalToHex function, which means that decimalToHex is effectively being called twice as often as it should.

This function changes the DOM a lot. Every time the loop runs, it adds a new div to the colors div tag. This makes me wonder whether that’s what the e.extend.merge function was doing. The profiler makes it easy to tell with a simple experiment.

Instead of adding a new div each time the loop runs, I want to add all of the div tags at once. Let’s create a variable to hold them, and then add them once at the end.

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { var colors = ""; for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); colors += '

'; } $('#colors').append(colors);}

This small change in the code means that the DOM changes once, when it adds all of the div tags. Testing that with the timeline, we see that the runtime between the click and the paint events is now 31 milliseconds. This one DOM change has brought the time for version four down by about 87%. We can also run the profiler again and see that the e.extend.merge function now takes up such a small percentage of the time that it doesn’t show up on the list.

We could make the code one notch faster by removing the decimalToHex function entirely. CSS supports RGB colors, so we don’t need to convert them to hex. Now we can write our makeColorSorter function like this:

function makeColorSorter(frequency1, frequency2, frequency3, phase1, phase2, phase3, center, width, len) { var colors = ""; for (var i = 0; i < len; ++i) { var red = Math.floor(Math.sin(frequency1 * i + phase1) * width + center); var green = Math.floor(Math.sin(frequency2 * i + phase2) * width + center); var blue = Math.floor(Math.sin(frequency3 * i + phase3) * width + center); colors += '

'; } $('#colors').append(colors);}

Version five runs in only 26 milliseconds and uses 18 lines of code for what used to take 28 lines.

Real-world applications are much more complex than this color sorter, but profiling them follows the same basic steps:

Establish a baseline so that you know where you’re starting from.Isolate the problem from any other code running in the application.Make it faster in a controlled environment, with frequent timelines and profiles.

There are a few other rules to follow when tuning performance:

Start with the slowest parts first so that you get the most improvement for the time spent tuning.Control the environment. If you switch computers or make any other major changes, always run a new baseline.Repeat the analysis to prevent anomalies on your computer from skewing the results.

Everyone wants their website to run faster. You have to develop new features, but new features usually make a website run slower. So, investing time in tuning the performance does pay off.

Profiling and tuning cut the final color sorter’s runtime by over 92%. How much faster could your website be?

(al) (km)

Zack Grossbart

Zack Grossbart is an engineer, designer, and author. He's a founding member of the Spiffy UI project, the architect of the WordPress Editorial Calendar, and a Consulting Engineer with NetIQ. Zack began loading DOS from a floppy disk when he was five years old. He first worked professionally with computers when he was 15 and started his first software company when he was 16. Zack lives in Cambridge, Massachusetts with his wife and daughter.

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

Read more >>

Tweet

Five great tools for troubleshooting your Mac

Takeaway: Wil Limoges shares the utilities he keeps in his support toolkit for troubleshooting Mac problems. What’s in yours?

Recently Forrester Research released a report stating, in a nutshell, that it’s time to start supporting the Mac in the enterprise. If you haven’t had an opportunity to read the specifics head over to Fortune’s Apple 2.0 blog and check out the brief version. If you’re interested in the full report it’s a bit pricy at $499 but you can purchase a copy directly from Forrester.

As more Macs make their way into organizations, large and small, here are some great support tools for maintaining and troubleshooting Macs.

Its usually pretty rare thing these days for a Mac to crash and lock up frequently, however, when and if it does happen memory is the first thing I think to test. Memtest OS X is great open source tool that runs from the command line for identifying issues within memory.

I’ve used DiskWarrior for years and have to come to rely on it as my primary data recovery tool. Instead of rebuilding damaged data on the faulty disk, DiskWarrior rebuilds the damaged drive based on the data, which can reduce imminent drive failure. It’s simple, powerful, and for m,e has been a reliable way to recover data.

These days I find myself relying primarily on Time Machine for backing up data. Time Machine, however, doesn’t always provide the flexiblility that is required for some tasks. This is where Carbon Copy Cloner comes in. Carbon Copy Cloner features the ability to back up to additional drives that Time Machine sometimes can’t access. It can be scheduled to back up at specific times, and is great for making a bootable backup of your existing drives.

TechTool Protogo is a great utility that you can take with you. Much like the aforementioned utilities, TechTool Protogo is great for scanning hard drives, testing hardware, backing up, and much more — only without the need to install it directly to a potentially unresponsive Mac. Just install the suite of tools on a USB thumb drive and take it with you.

iStat Pro is one of my favorite utilities. iStat is a simple widget that can be installed into Dashboard and is a powerful utility for accessing system information in a flash. It’s great for monitoring Wi-Fi signal strength, hardware temperatures, hard drive capacity and usage, and so much more. You can even install iStat Pro for iPhone and monitor your Macs remotely!

What are some of the tools that you like to use with Macs?

Wil Limoges is a Louisville, KY freelance web designer and Digital Savant at the vimarc group.

Read more >>

Tweet

How to add Disqus comment system to your blogger blog or website

DISQUS is a comments platform that helps you build an active community from your website's audience. It has awesome features, powerful tools, and it's easy to install.Before this I have explain how to add IntenseDebate comment system to blogger blog and now I am going to explain how to add Disqus comment system to your blogger blog.

There are lot of good features with Disqus comment system when we comparing it with Blogger default comment system.

Read more >>

Tweet

Read More